-- Logs begin at Mon 2022-10-31 14:14:44 UTC, end at Mon 2022-10-31 15:16:14 UTC. -- Oct 31 14:42:44.046296 dv-cinder-860227-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 31 14:42:44.057748 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 31 14:42:44.059831 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: *** Starting uWSGI 2.0.18-debian (64bit) on [Mon Oct 31 14:42:44 2022] *** Oct 31 14:42:44.059831 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 31 14:42:44.059831 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 31 14:42:44.059831 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: nodename: dv-cinder-860227-3-iscsi Oct 31 14:42:44.059831 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: machine: x86_64 Oct 31 14:42:44.059831 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: clock source: unix Oct 31 14:42:44.059831 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: pcre jit disabled Oct 31 14:42:44.059831 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: detected number of CPU cores: 4 Oct 31 14:42:44.059831 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: current working directory: / Oct 31 14:42:44.059831 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: detected binary path: /usr/bin/uwsgi-core Oct 31 14:42:44.059831 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: your processes number limit is 47680 Oct 31 14:42:44.059831 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: your memory page size is 4096 bytes Oct 31 14:42:44.059831 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: detected max file descriptor number: 2048 Oct 31 14:42:44.059831 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: lock engine: pthread robust mutexes Oct 31 14:42:44.060522 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: thunder lock: enabled Oct 31 14:42:44.060522 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 31 14:42:44.060522 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 31 14:42:44.100076 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: Python main interpreter initialized at 0x561deef029c0 Oct 31 14:42:44.100076 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: python threads support enabled Oct 31 14:42:44.100076 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: your server socket listen backlog is limited to 100 connections Oct 31 14:42:44.100076 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: your mercy for graceful operations on workers is 90 seconds Oct 31 14:42:44.100434 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: mapped 403077 bytes (393 KB) for 2 cores Oct 31 14:42:44.100520 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: *** Operational MODE: preforking *** Oct 31 14:42:44.100741 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: *** uWSGI is running in multiple interpreter mode *** Oct 31 14:42:44.100741 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: spawned uWSGI master process (pid: 96551) Oct 31 14:42:44.100932 dv-cinder-860227-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 31 14:42:44.103667 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: spawned uWSGI worker 1 (pid: 96564, cores: 1) Oct 31 14:42:44.103667 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: spawned uWSGI worker 2 (pid: 96565, cores: 1) Oct 31 14:42:44.103667 dv-cinder-860227-3-iscsi devstack@keystone.service[96551]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 31 14:42:48.423988 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 31 14:42:48.423988 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: from cryptography.utils import int_from_bytes Oct 31 14:42:48.425335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 31 14:42:48.425335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: from cryptography.utils import int_from_bytes Oct 31 14:42:48.427998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 31 14:42:48.427998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: from cryptography.utils import int_from_bytes Oct 31 14:42:48.428474 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 31 14:42:48.428474 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: from cryptography.utils import int_from_bytes Oct 31 14:42:48.474576 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 31 14:42:48.474760 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 31 14:42:48.474951 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] command line args: [] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 31 14:42:48.475211 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 31 14:42:48.475380 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ================================================================================ {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 31 14:42:48.475669 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] admin_token = **** {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.475919 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] config_dir = [] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.476110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.476192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 31 14:42:48.476334 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] config_source = [] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.476414 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 31 14:42:48.476482 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] debug = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.476541 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] command line args: [] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 31 14:42:48.476711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.476909 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 31 14:42:48.476980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ================================================================================ {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 31 14:42:48.478059 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] admin_token = **** {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.478274 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.478274 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] insecure_debug = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.478697 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] config_dir = [] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.478697 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.479049 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.479277 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] config_source = [] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.479456 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.479717 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] debug = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.479923 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] list_limit = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.480091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.480349 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] log_config_append = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.480551 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.480754 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.480959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] insecure_debug = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.481133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] log_dir = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.481334 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.481563 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] log_file = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.481563 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.481930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.481930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] list_limit = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.482315 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.482565 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] log_config_append = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.482769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.482938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.483138 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.483316 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] log_dir = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.483485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.483643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] log_file = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.483773 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.483937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.484083 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.484224 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.484352 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.484480 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.484642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.484776 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.484916 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.485044 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.485044 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.485306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.485306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.485561 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.485697 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.485858 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.485858 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.486258 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] notification_format = cadf {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.486397 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.486526 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.486674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.486674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.486926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.36/identity {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.487669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.488231 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] publish_errors = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.488368 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] notification_format = cadf {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.488496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.488496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.488822 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.488961 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.488961 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.489219 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.36/identity {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.489348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.489481 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] publish_errors = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.489618 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] standard_threads = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.489753 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.489899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] strict_password_check = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.490027 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.490167 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.490295 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.490424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] use_eventlog = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.490567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.490702 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] use_journal = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.490830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.491026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] use_json = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.491026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] use_stderr = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.491280 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] standard_threads = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.491409 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] use_syslog = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.491544 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] strict_password_check = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.491719 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] watch_log_file = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.491861 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.491991 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.492118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] use_eventlog = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.492118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] use_journal = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.492375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.492563 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] use_json = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.492710 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.492840 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] use_stderr = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.492968 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.493102 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] use_syslog = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.493239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.493368 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] watch_log_file = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 31 14:42:48.493504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.493639 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.493769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.493896 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.494024 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.494151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.494302 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.494433 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.494561 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.494688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.494815 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.494949 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.495086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.495215 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.495344 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.495478 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.495638 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.495779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.495993 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.496141 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.496281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.496413 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.496547 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.496676 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.496813 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.496942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.497070 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.497204 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.497338 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.497468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.497672 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.497803 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.497931 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.498075 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.498205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.498341 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.498471 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.498600 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.498728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.498856 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.498983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.499118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.499247 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.499374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.499508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.499696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.499827 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.499977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.500113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.500248 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.500377 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.500504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.500633 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.500768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.500903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.501031 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.501171 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.501301 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.501428 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.501563 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.501709 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.501847 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.501975 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.502103 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.502230 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.502365 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.502504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.503400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.503913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.504053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.504184 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.504322 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.504452 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.504586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.504727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.504857 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.504990 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.505199 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.505330 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.505466 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.505594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.505722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.505850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.505986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.506114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.506242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.506375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.506782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.507186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.507333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.507725 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.507928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.508125 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.508320 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.508514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.508754 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.508920 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.509116 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.509310 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.509504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.509698 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.509942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.510111 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.510305 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.510531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.510701 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.510932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.511164 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.511368 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.511536 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.511738 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.511971 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.512183 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.512350 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.512556 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.512783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.512951 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.513086 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.513260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.513403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.513565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.513698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.513840 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.513971 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.514108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.514271 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.514414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.514567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.514698 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.514827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.514956 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.515090 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.515229 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.515360 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.515488 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.515704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.515836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.515965 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.516105 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.516244 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.516375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.enabled = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.516375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.516631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.516766 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.516907 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.517037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.517166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.517294 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.517423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.517552 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.517689 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.517819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.517947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.enabled = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.518075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.518772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.518924 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.518924 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.518924 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.518924 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.519414 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.519545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.519700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.519843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.519993 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.520240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.520399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.520542 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.520673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.520822 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.520953 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.521088 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.521218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.521346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.521476 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.521687 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.521818 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.521948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.521948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.522192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.522321 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.522461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.522594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.522734 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.522922 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.523070 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.523213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.523343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.523487 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.523644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.523776 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.523906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.524036 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.524164 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.524305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.524436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.524565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.524714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.524845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.524982 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.525120 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] auth.external = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.525250 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.525379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] auth.mapped = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.525508 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.525508 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.525760 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.526228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.526372 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.526503 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.526662 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] auth.password = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.526662 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] auth.token = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.526936 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.527066 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.527196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.527335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] catalog.caching = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.527465 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.527636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.527775 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.527907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.528037 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.528176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.528308 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] auth.external = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.528438 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.528573 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] auth.mapped = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.528713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.528844 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.528988 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] credential.caching = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.529119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.529248 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] credential.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.529377 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] auth.password = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.529514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.529645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] auth.token = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.529784 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.529916 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.530051 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.530182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] catalog.caching = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.530311 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.530462 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.530602 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.530733 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.530863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.530993 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.531124 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.531254 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.531397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.531528 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.531684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.531825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] credential.caching = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.531957 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.532088 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] credential.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.532227 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.532370 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.532508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.532639 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.532769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.532937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.533085 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.533218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.533360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] federation.caching = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.533492 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.533630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.533762 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.533901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] federation.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.534560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.534719 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.534854 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.534995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.535128 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.535269 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.535402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.535533 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.535711 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.535843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.536031 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.536170 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.536303 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.536439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.536571 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.536713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.536852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] federation.caching = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.536985 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.537115 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.537245 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] identity.caching = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.537377 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] federation.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.537507 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.537645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.537778 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.537909 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.538078 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.538225 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.538360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.538578 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.538711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] identity.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.538843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.538981 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.539113 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.539243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.539382 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.539520 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.539680 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.539813 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.539944 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.540074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.540216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] identity.caching = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.540349 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.540492 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.540623 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.540754 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.540885 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.541023 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.541155 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.541286 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.541416 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.541547 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] identity.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.541677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.541816 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.541948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.542085 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.542226 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.542357 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.542494 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.542635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.542767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.542897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.543064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.543201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.543332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.543471 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.543674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.543845 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.543983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.544115 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.544259 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.544402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.544535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.544666 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.544797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.544927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.545057 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.545195 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.545327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.545457 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.545586 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.545586 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.545833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.545833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.546097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.546234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.546366 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.546495 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.546633 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.546766 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.546896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.547026 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.547162 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.password = **** {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.547162 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.547421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.547588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.547729 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.547872 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.548002 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.548132 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.548278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.548278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.548525 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.548654 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.548785 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.548915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.549052 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.549183 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.549314 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.549941 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.550251 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.550707 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.550853 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.550988 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.password = **** {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.551119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.551251 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.551381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.551512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.551704 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.551838 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.551971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.552101 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.552232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.552362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.552511 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.552643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.552774 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.552905 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.553037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.553209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.553349 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.553481 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.553612 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.553741 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.553882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.554012 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.554152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.554283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.554424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.554555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.554685 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.554685 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.555715 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.555861 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.555994 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.556125 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.556255 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.556385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.556524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.556656 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.556787 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.556917 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.557059 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.557190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.557334 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.557465 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.557595 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.557724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.557854 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.557983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.558122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.558302 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.558438 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.558570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.558700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.558840 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.558980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.559111 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.559242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.559374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.559504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.559656 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.559800 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.559932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.560063 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.560745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] policy.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.560893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.561026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.561167 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.561298 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.561429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.561568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.561699 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.561829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] receipt.caching = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.561959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.562089 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.562226 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.562365 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.562499 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.562629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.562759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.562889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.563029 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.563216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.563355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.563488 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] resource.caching = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.563648 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.563781 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.563911 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] policy.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.564053 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] resource.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.564187 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.564317 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.564448 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.564579 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.564710 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.564849 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.564992 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] receipt.caching = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.565712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] revoke.caching = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.566433 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.566594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.566732 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.566874 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.567007 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.567138 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] role.cache_time = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.567268 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.567398 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] role.caching = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.567529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.567780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] role.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.567924 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] resource.caching = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.568067 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] role.list_limit = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.568198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.568329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.568464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] resource.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.568605 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.568745 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.568877 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.569008 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.569138 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.569277 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.569409 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.569539 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] revoke.caching = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.569669 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.569801 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.569940 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.570080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.570211 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.570342 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] role.cache_time = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.570472 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.570602 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] role.caching = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.570731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.570876 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] role.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.571046 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.571183 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] role.list_limit = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.571313 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.571443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.571600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.571762 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.571762 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.572014 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.572014 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.572259 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.572389 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.572528 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.572666 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.572797 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.572929 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.573065 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.573195 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.573376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.573520 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.573651 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.573781 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.573921 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.573921 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.574189 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.574320 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.574450 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.574580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.574713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.574713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.574968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.575102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.575233 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.575373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.575503 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.575691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.575789 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.575868 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.575939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.576011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.576083 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.576165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.576239 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.576311 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] token.cache_time = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.576383 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.576455 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] token.caching = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.576455 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.576593 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.576667 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] token.provider = fernet {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.576739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.576810 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.576880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.576880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.577017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.577092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.577163 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.577234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.577305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.577305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.577440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.577513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.577584 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.577655 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] trust.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.577726 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.577796 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.577875 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.577948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.578020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] token.cache_time = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.578091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.578162 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] token.caching = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.578233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.578311 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.578384 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.578455 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] token.provider = fernet {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.578525 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.578605 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.578686 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.578759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.578838 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.578912 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.578982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.connection = **** {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.579053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.579123 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.579195 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.579277 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.579352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.579423 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.579494 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] trust.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.579595 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.579671 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.579756 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.579831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.579901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.579972 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.580042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.580113 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.580191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.580273 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.580346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.580423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.580503 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.580584 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.580660 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.580861 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.580937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581001 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.connection = **** {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581059 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581183 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581313 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581490 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581548 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581672 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581672 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=96564) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 31 14:42:48.581814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.581814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.582305 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.582305 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.582305 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.582305 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 31 14:42:48.582305 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=96565) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 31 14:42:48.591587 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.591871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.592024 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.601044 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.601279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.601503 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.602743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.609901 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.632182 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.632501 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.638431 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.638666 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.687800 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.688060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.740380 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.740684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.740684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.740982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.897410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.897677 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.897868 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.898093 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.898314 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.898398 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.898531 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.898804 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.899129 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.899452 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.899819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.900139 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.922424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.922659 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.922840 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.922942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.923090 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.923307 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.923388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.923521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.923759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.923899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.924225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.924610 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.924918 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.925239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.935191 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.935796 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.948630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.948747 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.948867 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.949103 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.949197 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.949297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.949512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.949592 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.949725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.949940 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=96565) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.950092 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.950411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.950729 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.951046 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=96564) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 31 14:42:48.966871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.967083 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 31 14:42:48.967166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.967433 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.967513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 31 14:42:48.967667 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 31 14:42:48.967947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.968132 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.968213 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.968494 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 31 14:42:48.968725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.968975 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.969057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.969405 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.969768 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.970128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.979169 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.979456 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.979756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.980044 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.981066 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.981456 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.981832 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.982233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.987043 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:48.989527 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:48.989954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:48.990242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.990530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.990818 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.991107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.991372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.991666 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.991944 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.992335 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.992619 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.992895 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.993646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:48.994038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.994425 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.994812 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.995199 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.995586 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.995967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.996361 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.996741 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.997135 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:48.997506 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.015849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.017317 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.017607 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.021165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.022766 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.023169 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.024391 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.024687 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.024974 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.028305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.028705 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.029099 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.031446 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.031763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.036454 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.036563 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.036850 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.036934 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.037149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.037447 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.037731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.038017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.038289 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.038569 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.047840 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.048152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.048462 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.048750 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.049110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.049426 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.049747 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.050023 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.054344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.054635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.058506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.061776 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.062079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.066380 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.066660 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.066843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.069974 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.070515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.073202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.073505 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.073793 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.074093 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.074237 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.074411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.074558 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.074706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.078652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.082161 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.085365 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.085671 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.085959 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.086258 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.086555 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.086891 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.090663 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.090950 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.091221 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.091490 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.099139 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.100730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.102284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.102566 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.104470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.104768 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.105040 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.105285 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.105373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.105577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.105973 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.106261 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.106556 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.112635 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.114602 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.116159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.116439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.118427 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.118731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.119120 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.119216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.119410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.119507 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.119723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.119810 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.120008 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.120096 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.120291 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.120377 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.131791 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.132168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.132454 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.132736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.133133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.133416 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.133695 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.135644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.138533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.141428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.142952 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.144883 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.145173 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.145460 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.149151 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.151786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.152043 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.153265 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.154903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.156342 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.156637 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.156725 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.156927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.157211 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.158114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.158392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.159330 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.165663 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.165939 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.166029 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.166234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.167393 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.170343 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.170742 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.171032 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.171314 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.174160 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=96565) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.175028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.175288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.175581 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.175861 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.176138 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.176415 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.176706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.176979 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.177250 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.177534 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.177806 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.178072 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.178408 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=96565) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.179788 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.180082 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.180365 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.187196 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=96564) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 31 14:42:49.187482 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.187790 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.188070 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.188348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.188627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.188902 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.189196 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.189479 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.189749 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.190042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.191506 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.191843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.192135 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=96564) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 31 14:42:49.212000 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x561deef029c0 pid: 96565 (default app) Oct 31 14:42:49.215381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af39f57-d5cf-45c8-bb3c-beaf7aeba61c None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:42:49.215703 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af39f57-d5cf-45c8-bb3c-beaf7aeba61c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:42:49.215978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af39f57-d5cf-45c8-bb3c-beaf7aeba61c None None] PATH_INFO: `/v3` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:42:49.220943 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1/1] 10.222.0.36 () {50 vars in 791 bytes} [Mon Oct 31 14:42:49 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:42:49.232500 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x561deef029c0 pid: 96564 (default app) Oct 31 14:42:56.058351 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562c4377-a69a-4f2e-850f-27e8f476d860 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:42:56.058964 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562c4377-a69a-4f2e-850f-27e8f476d860 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:42:56.058964 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562c4377-a69a-4f2e-850f-27e8f476d860 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:42:56.060216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 2/2] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:42:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:42:56.068017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd096267-eccf-4132-96d8-33ed45e3901c None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:42:56.068514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd096267-eccf-4132-96d8-33ed45e3901c None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:42:56.068650 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd096267-eccf-4132-96d8-33ed45e3901c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:42:56.139873 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: INFO dbcounter [None req-bd096267-eccf-4132-96d8-33ed45e3901c None None] Registered counter for database keystone Oct 31 14:42:56.160078 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG oslo_db.sqlalchemy.engines [None req-bd096267-eccf-4132-96d8-33ed45e3901c None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=96564) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:326}} Oct 31 14:42:56.166879 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writer thread running {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 31 14:42:56.346776 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG passlib.handlers.bcrypt [None req-bd096267-eccf-4132-96d8-33ed45e3901c None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=96564) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 31 14:42:56.346776 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG passlib.handlers.bcrypt [None req-bd096267-eccf-4132-96d8-33ed45e3901c None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=96564) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 31 14:42:56.405172 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-bd096267-eccf-4132-96d8-33ed45e3901c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:42:56.406771 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-bd096267-eccf-4132-96d8-33ed45e3901c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:42:56.535981 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1/3] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:42:56 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 471 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 31 14:42:56.977149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562c4377-a69a-4f2e-850f-27e8f476d860 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:42:57.046567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: INFO dbcounter [None req-562c4377-a69a-4f2e-850f-27e8f476d860 None None] Registered counter for database keystone Oct 31 14:42:57.068849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG oslo_db.sqlalchemy.engines [None req-562c4377-a69a-4f2e-850f-27e8f476d860 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=96565) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:326}} Oct 31 14:42:57.074988 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writer thread running {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 31 14:42:57.193415 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562c4377-a69a-4f2e-850f-27e8f476d860 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:42:57.197759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b848d1-d6da-47ac-af63-b3f474774676 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:42:57.198378 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b848d1-d6da-47ac-af63-b3f474774676 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:42:57.198660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b848d1-d6da-47ac-af63-b3f474774676 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:42:57.198932 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b848d1-d6da-47ac-af63-b3f474774676 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:42:57.207133 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b848d1-d6da-47ac-af63-b3f474774676 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:42:57.259308 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: WARNING keystone.common.rbac_enforcer.enforcer [None req-f5b848d1-d6da-47ac-af63-b3f474774676 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 31 14:42:57.259620 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b848d1-d6da-47ac-af63-b3f474774676 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:42:57.269074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 3/4] 10.222.0.36 () {58 vars in 1324 bytes} [Mon Oct 31 14:42:56 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 31 14:42:57.275280 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd096267-eccf-4132-96d8-33ed45e3901c None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:42:57.296130 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd096267-eccf-4132-96d8-33ed45e3901c None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:42:57.301316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01023f5-0f47-4d8c-981e-9a627d6679aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:42:57.302609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01023f5-0f47-4d8c-981e-9a627d6679aa admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:42:57.302891 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01023f5-0f47-4d8c-981e-9a627d6679aa admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:42:57.303158 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01023f5-0f47-4d8c-981e-9a627d6679aa admin admin] PATH_INFO: `/v3/projects/e18b2b57837347e48db3adc0214764b8` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:42:57.304627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01023f5-0f47-4d8c-981e-9a627d6679aa admin admin] RBAC: Authorizing `identity:get_project(project_id=e18b2b57837347e48db3adc0214764b8)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:42:57.356499 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: WARNING keystone.common.rbac_enforcer.enforcer [None req-d01023f5-0f47-4d8c-981e-9a627d6679aa admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 31 14:42:57.356836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01023f5-0f47-4d8c-981e-9a627d6679aa admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:42:57.361291 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 2/5] 10.222.0.36 () {56 vars in 1207 bytes} [Mon Oct 31 14:42:57 2022] GET /identity/v3/projects/e18b2b57837347e48db3adc0214764b8 => generated 344 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:42:59.500785 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df669a46-c06a-4ef4-841e-d8daf65447d7 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:42:59.501323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df669a46-c06a-4ef4-841e-d8daf65447d7 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:42:59.501323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df669a46-c06a-4ef4-841e-d8daf65447d7 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:42:59.505959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 4/6] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:42:59 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:42:59.511346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68f28c8-3c92-487a-b114-527eb11f237f None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:42:59.511852 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68f28c8-3c92-487a-b114-527eb11f237f None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:42:59.511942 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68f28c8-3c92-487a-b114-527eb11f237f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:42:59.532488 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-e68f28c8-3c92-487a-b114-527eb11f237f None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:42:59.534119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e68f28c8-3c92-487a-b114-527eb11f237f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:42:59.542002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 3/7] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:42:59 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 31 14:42:59.811586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df669a46-c06a-4ef4-841e-d8daf65447d7 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:42:59.824338 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df669a46-c06a-4ef4-841e-d8daf65447d7 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:42:59.828498 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca6bdfb-4921-4816-81f5-1976df0df94e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:42:59.829089 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca6bdfb-4921-4816-81f5-1976df0df94e admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:42:59.829367 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca6bdfb-4921-4816-81f5-1976df0df94e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:42:59.829637 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca6bdfb-4921-4816-81f5-1976df0df94e admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:42:59.837296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca6bdfb-4921-4816-81f5-1976df0df94e admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:42:59.839050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca6bdfb-4921-4816-81f5-1976df0df94e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:42:59.847832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 5/8] 10.222.0.36 () {58 vars in 1324 bytes} [Mon Oct 31 14:42:59 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 31 14:42:59.853215 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68f28c8-3c92-487a-b114-527eb11f237f None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:42:59.862960 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68f28c8-3c92-487a-b114-527eb11f237f None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:42:59.867370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817bda71-a0f9-419c-ad41-874e3ad69c02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:42:59.868022 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817bda71-a0f9-419c-ad41-874e3ad69c02 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:42:59.868310 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817bda71-a0f9-419c-ad41-874e3ad69c02 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:42:59.868619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817bda71-a0f9-419c-ad41-874e3ad69c02 admin admin] PATH_INFO: `/v3/users/7c12195059ec40bfafbb7be8e9f46734` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:42:59.869946 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817bda71-a0f9-419c-ad41-874e3ad69c02 admin admin] RBAC: Authorizing `identity:get_user(user_id=7c12195059ec40bfafbb7be8e9f46734)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:42:59.871809 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817bda71-a0f9-419c-ad41-874e3ad69c02 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:42:59.873350 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 4/9] 10.222.0.36 () {56 vars in 1198 bytes} [Mon Oct 31 14:42:59 2022] GET /identity/v3/users/7c12195059ec40bfafbb7be8e9f46734 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:02.063886 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7369b5b-eb50-4685-8797-b71c5be99dbc None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:02.064508 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7369b5b-eb50-4685-8797-b71c5be99dbc None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:02.064508 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7369b5b-eb50-4685-8797-b71c5be99dbc None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:02.065871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 6/10] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:43:02.070994 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0bad4d-e52e-4ba9-b8c4-391d444b54b7 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:02.071370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0bad4d-e52e-4ba9-b8c4-391d444b54b7 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:02.071690 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0bad4d-e52e-4ba9-b8c4-391d444b54b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:02.090498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-8f0bad4d-e52e-4ba9-b8c4-391d444b54b7 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:02.091705 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-8f0bad4d-e52e-4ba9-b8c4-391d444b54b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:02.130303 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c789e22f-7e4f-47de-ac25-5fc5b9d85787 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:02.130586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c789e22f-7e4f-47de-ac25-5fc5b9d85787 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:02.130865 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c789e22f-7e4f-47de-ac25-5fc5b9d85787 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:02.132218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 7/11] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:43:02.137360 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf04e70e-5d0a-4aa5-b2c9-b050dfbd0a1e None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:02.137642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf04e70e-5d0a-4aa5-b2c9-b050dfbd0a1e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:02.137914 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf04e70e-5d0a-4aa5-b2c9-b050dfbd0a1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:02.174593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 5/12] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:02.180871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG passlib.handlers.bcrypt [None req-cf04e70e-5d0a-4aa5-b2c9-b050dfbd0a1e None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=96565) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 31 14:43:02.181266 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG passlib.handlers.bcrypt [None req-cf04e70e-5d0a-4aa5-b2c9-b050dfbd0a1e None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=96565) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 31 14:43:02.230869 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-cf04e70e-5d0a-4aa5-b2c9-b050dfbd0a1e None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:02.232480 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-cf04e70e-5d0a-4aa5-b2c9-b050dfbd0a1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:02.331949 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 8/13] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:43:02.444770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0bad4d-e52e-4ba9-b8c4-391d444b54b7 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:02.471252 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:02.551889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0bad4d-e52e-4ba9-b8c4-391d444b54b7 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:02.586697 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0adeff0d-2e07-4a37-8fda-8b459b376eb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:02.587264 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adeff0d-2e07-4a37-8fda-8b459b376eb7 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:02.587543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adeff0d-2e07-4a37-8fda-8b459b376eb7 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:02.587844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adeff0d-2e07-4a37-8fda-8b459b376eb7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:02.599166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0adeff0d-2e07-4a37-8fda-8b459b376eb7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:02.601612 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf04e70e-5d0a-4aa5-b2c9-b050dfbd0a1e None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:02.608979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-0adeff0d-2e07-4a37-8fda-8b459b376eb7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:43:02.608979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:02.608979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:02.608979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:02.608979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:02.608979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:02.608979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:02.608979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:02.608979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:02.608979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:02.608979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:02.608979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:02.608979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:02.608979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:02.608979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:02.608979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:02.610006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:02.610006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:02.610006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:02.610006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:02.610006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:02.610006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:02.610006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:02.610006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:02.610006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:02.610006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:02.610006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:02.610006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:02.610006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:02.610006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:02.610006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:02.610006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:02.610830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:02.610830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:02.610830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:02.610830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:02.610830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:02.610830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:02.610830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:02.610830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:02.610830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:02.610830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:02.610830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:43:02.610830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:02.610830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 6/14] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:43:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:02.616011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0adeff0d-2e07-4a37-8fda-8b459b376eb7 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:02.697023 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:02.702581 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf04e70e-5d0a-4aa5-b2c9-b050dfbd0a1e None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:02.705056 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:02.715821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0adeff0d-2e07-4a37-8fda-8b459b376eb7 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:02.741855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6da6f26-1a7d-4cd4-b61b-f180d508fa8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:02.742719 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6da6f26-1a7d-4cd4-b61b-f180d508fa8d None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:02.743008 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6da6f26-1a7d-4cd4-b61b-f180d508fa8d None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:02.743282 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6da6f26-1a7d-4cd4-b61b-f180d508fa8d None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:02.744283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6da6f26-1a7d-4cd4-b61b-f180d508fa8d None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:02.745724 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6da6f26-1a7d-4cd4-b61b-f180d508fa8d None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:02.750338 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d15798-e49a-4cde-8307-ddef6763906b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:02.750945 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d15798-e49a-4cde-8307-ddef6763906b None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:02.751231 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d15798-e49a-4cde-8307-ddef6763906b None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:02.751509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d15798-e49a-4cde-8307-ddef6763906b None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:02.752528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d15798-e49a-4cde-8307-ddef6763906b None admin] RBAC: Adding query filter params (name=admin) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:02.752841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d15798-e49a-4cde-8307-ddef6763906b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:02.754232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d15798-e49a-4cde-8307-ddef6763906b None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:02.756806 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.sql.core [None req-c6da6f26-1a7d-4cd4-b61b-f180d508fa8d None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 31 14:43:02.756806 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 31 14:43:02.756806 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [parameters: {'id': 'd2c0f4d4b9014de5b138565d04af293d', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 31 14:43:02.756806 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=96565) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 31 14:43:02.771226 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 7/15] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:02.772279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d15798-e49a-4cde-8307-ddef6763906b None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:02.776906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-c6da6f26-1a7d-4cd4-b61b-f180d508fa8d None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 31 14:43:02.776906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:02.776906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 31 14:43:02.776906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 31 14:43:02.776906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 31 14:43:02.776906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 31 14:43:02.776906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 31 14:43:02.776906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application result = self._query(query) Oct 31 14:43:02.776906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 31 14:43:02.776906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application conn.query(q) Oct 31 14:43:02.776906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 31 14:43:02.776906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 31 14:43:02.776906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 31 14:43:02.776906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application result.read() Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 31 14:43:02.777751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 31 14:43:02.778642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application next(self.gen) Oct 31 14:43:02.778642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 31 14:43:02.778642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application yield resource Oct 31 14:43:02.778642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 31 14:43:02.778642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application next(self.gen) Oct 31 14:43:02.778642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 31 14:43:02.778642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self.session.rollback() Oct 31 14:43:02.778642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 31 14:43:02.778642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self.force_reraise() Oct 31 14:43:02.778642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 31 14:43:02.778642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise self.value Oct 31 14:43:02.778642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 31 14:43:02.778642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 31 14:43:02.778642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 31 14:43:02.778642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application session.commit() Oct 31 14:43:02.778642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 31 14:43:02.778642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 31 14:43:02.779441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 31 14:43:02.779441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._prepare_impl() Oct 31 14:43:02.779441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 31 14:43:02.779441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self.session.flush() Oct 31 14:43:02.779441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 31 14:43:02.779441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._flush(objects) Oct 31 14:43:02.779441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 31 14:43:02.779441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 31 14:43:02.779441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 31 14:43:02.779441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application compat.raise_( Oct 31 14:43:02.779441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 31 14:43:02.779441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception Oct 31 14:43:02.779441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 31 14:43:02.779441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application flush_context.execute() Oct 31 14:43:02.779441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 31 14:43:02.779441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rec.execute(self) Oct 31 14:43:02.779441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 31 14:43:02.780343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 31 14:43:02.780343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 31 14:43:02.780343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 31 14:43:02.780343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 31 14:43:02.780343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 31 14:43:02.780343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 31 14:43:02.780343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 31 14:43:02.780343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 31 14:43:02.780343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 31 14:43:02.780343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 31 14:43:02.780343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 31 14:43:02.780343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 31 14:43:02.780343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 31 14:43:02.780343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 31 14:43:02.780343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 31 14:43:02.781092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 31 14:43:02.781092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception Oct 31 14:43:02.781092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 31 14:43:02.781092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 31 14:43:02.781092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 31 14:43:02.781092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 31 14:43:02.781092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 31 14:43:02.781092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application result = self._query(query) Oct 31 14:43:02.781092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 31 14:43:02.781092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application conn.query(q) Oct 31 14:43:02.781092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 31 14:43:02.781092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 31 14:43:02.781092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 31 14:43:02.781092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application result.read() Oct 31 14:43:02.781092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 31 14:43:02.781092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 31 14:43:02.781092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 31 14:43:02.781893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 31 14:43:02.781893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 31 14:43:02.781893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 31 14:43:02.781893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 31 14:43:02.781893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 31 14:43:02.781893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 31 14:43:02.781893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 31 14:43:02.781893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [parameters: {'id': 'd2c0f4d4b9014de5b138565d04af293d', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 31 14:43:02.781893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 31 14:43:02.781893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:02.781893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:43:02.781893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:02.781893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:02.781893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 31 14:43:02.781893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 31 14:43:02.782606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 31 14:43:02.782606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 31 14:43:02.782606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 31 14:43:02.782606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:02.782606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:43:02.782606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:02.782606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:02.782606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:02.782606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:02.782606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:02.782606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:02.782606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:02.782606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:02.782606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:02.782606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:02.782606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:02.782606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:02.784252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 31 14:43:02.784252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 31 14:43:02.784252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:02.784252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:02.784252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 31 14:43:02.784252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 31 14:43:02.784252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:02.784252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:02.784252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 31 14:43:02.784252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 31 14:43:02.784252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 31 14:43:02.784252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:02.784252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 9/16] 10.222.0.36 () {60 vars in 1137 bytes} [Mon Oct 31 14:43:02 2022] POST /identity/v3/domains => generated 190 bytes in 177 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:43:02.786626 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6da6f26-1a7d-4cd4-b61b-f180d508fa8d None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:02.874822 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d15798-e49a-4cde-8307-ddef6763906b None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:02.884315 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6da6f26-1a7d-4cd4-b61b-f180d508fa8d None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:02.908703 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7db58c0-4ebb-48f7-bc19-e79de36ec367 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:02.909335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7db58c0-4ebb-48f7-bc19-e79de36ec367 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:02.909610 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7db58c0-4ebb-48f7-bc19-e79de36ec367 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:02.909884 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7db58c0-4ebb-48f7-bc19-e79de36ec367 None admin] PATH_INFO: `/v3/users/7c12195059ec40bfafbb7be8e9f46734` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:02.911040 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7db58c0-4ebb-48f7-bc19-e79de36ec367 None admin] RBAC: Authorizing `identity:get_user(user_id=7c12195059ec40bfafbb7be8e9f46734)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:02.912537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7db58c0-4ebb-48f7-bc19-e79de36ec367 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:02.914101 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 8/17] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:02 2022] GET /identity/v3/users/7c12195059ec40bfafbb7be8e9f46734 => generated 247 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:02.919431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7db58c0-4ebb-48f7-bc19-e79de36ec367 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:02.920206 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13207aa4-8b88-4d37-b42b-fbebf63e3582 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:02.921065 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13207aa4-8b88-4d37-b42b-fbebf63e3582 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:02.921348 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13207aa4-8b88-4d37-b42b-fbebf63e3582 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:02.921624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13207aa4-8b88-4d37-b42b-fbebf63e3582 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:02.934820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:02.940475 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13207aa4-8b88-4d37-b42b-fbebf63e3582 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:02.941937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13207aa4-8b88-4d37-b42b-fbebf63e3582 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:02.958862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-13207aa4-8b88-4d37-b42b-fbebf63e3582 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:43:02.958862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:02.958862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:02.958862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:02.958862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:02.958862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:02.958862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:02.958862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:02.958862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:02.958862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:02.958862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:02.958862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:02.958862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:43:02.958862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:43:02.958862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:43:02.958862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:43:02.959760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:02.959760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:02.959760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:02.959760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:02.959760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:02.959760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:02.959760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:02.959760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:02.959760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:02.959760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:02.959760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:02.959760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:02.959760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:02.959760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:02.959760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:02.959760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:02.959760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:43:02.960555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:43:02.960555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:43:02.960555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:02.960555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 10/18] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:43:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:43:02.965766 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13207aa4-8b88-4d37-b42b-fbebf63e3582 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:02.990652 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:03.022375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7db58c0-4ebb-48f7-bc19-e79de36ec367 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:03.055918 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e86645-8170-4c03-82df-5c0eaea44bed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:03.056550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e86645-8170-4c03-82df-5c0eaea44bed None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:03.056833 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e86645-8170-4c03-82df-5c0eaea44bed None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:03.057110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e86645-8170-4c03-82df-5c0eaea44bed None admin] PATH_INFO: `/v3/domains/default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:03.058236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e86645-8170-4c03-82df-5c0eaea44bed None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:03.059704 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e86645-8170-4c03-82df-5c0eaea44bed None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:03.061263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 9/19] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:43:02 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:03.067226 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e86645-8170-4c03-82df-5c0eaea44bed None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:03.069129 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13207aa4-8b88-4d37-b42b-fbebf63e3582 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:03.103162 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1965ff3-bb19-4e51-9e2f-3e4d3c81de00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:03.103801 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1965ff3-bb19-4e51-9e2f-3e4d3c81de00 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:03.104087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1965ff3-bb19-4e51-9e2f-3e4d3c81de00 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:03.104362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1965ff3-bb19-4e51-9e2f-3e4d3c81de00 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:03.105006 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1965ff3-bb19-4e51-9e2f-3e4d3c81de00 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:03.105310 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1965ff3-bb19-4e51-9e2f-3e4d3c81de00 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:03.106696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1965ff3-bb19-4e51-9e2f-3e4d3c81de00 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:03.123633 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 11/20] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:43:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:03.151174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:03.172386 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e86645-8170-4c03-82df-5c0eaea44bed None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:03.211036 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9926432d-719b-4df3-bc44-287a26ec64f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:03.211739 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9926432d-719b-4df3-bc44-287a26ec64f0 None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:03.212027 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9926432d-719b-4df3-bc44-287a26ec64f0 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:03.212303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9926432d-719b-4df3-bc44-287a26ec64f0 None admin] PATH_INFO: `/v3/domains/default/users/7c12195059ec40bfafbb7be8e9f46734/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:03.214281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9926432d-719b-4df3-bc44-287a26ec64f0 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=7c12195059ec40bfafbb7be8e9f46734, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:03.215786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9926432d-719b-4df3-bc44-287a26ec64f0 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:03.335045 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 10/21] 10.222.0.36 () {58 vars in 1361 bytes} [Mon Oct 31 14:43:03 2022] PUT /identity/v3/domains/default/users/7c12195059ec40bfafbb7be8e9f46734/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:43:05.910777 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dbca18-4219-434f-ab0a-7f2cc58fb3ac None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:05.911292 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dbca18-4219-434f-ab0a-7f2cc58fb3ac None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:05.911366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dbca18-4219-434f-ab0a-7f2cc58fb3ac None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:05.913134 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 12/22] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:05.926373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56e2b92-155e-460d-971c-6863e59e062c None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:05.926833 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56e2b92-155e-460d-971c-6863e59e062c None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:05.926924 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56e2b92-155e-460d-971c-6863e59e062c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:05.953969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-a56e2b92-155e-460d-971c-6863e59e062c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:05.956725 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a56e2b92-155e-460d-971c-6863e59e062c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:06.040697 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 11/23] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:43:06.110601 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f89c386-8b2d-4363-bf8b-1d2e82969168 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:06.110932 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f89c386-8b2d-4363-bf8b-1d2e82969168 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:06.111191 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f89c386-8b2d-4363-bf8b-1d2e82969168 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:06.112822 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 13/24] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:06.118367 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4196d8ac-976b-4079-b664-43e9aad088bb None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:06.118705 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4196d8ac-976b-4079-b664-43e9aad088bb None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:06.118971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4196d8ac-976b-4079-b664-43e9aad088bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:06.144575 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-4196d8ac-976b-4079-b664-43e9aad088bb None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:06.145780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-4196d8ac-976b-4079-b664-43e9aad088bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:06.196493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d248b86-7d13-4445-a171-b81a5bc4b507 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:06.196781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d248b86-7d13-4445-a171-b81a5bc4b507 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:06.196975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d248b86-7d13-4445-a171-b81a5bc4b507 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:06.198428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 14/25] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:06.204129 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c90d694-c0bd-458c-ae61-3f2d7cb9295a None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:06.204437 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c90d694-c0bd-458c-ae61-3f2d7cb9295a None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:06.204837 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c90d694-c0bd-458c-ae61-3f2d7cb9295a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:06.230076 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:06.238078 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 12/26] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:06.241535 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-3c90d694-c0bd-458c-ae61-3f2d7cb9295a None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:06.242870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-3c90d694-c0bd-458c-ae61-3f2d7cb9295a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:06.334263 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 15/27] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:06.508838 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4196d8ac-976b-4079-b664-43e9aad088bb None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:06.546130 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c90d694-c0bd-458c-ae61-3f2d7cb9295a None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:06.562211 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:06.705487 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4196d8ac-976b-4079-b664-43e9aad088bb None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:06.728920 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c90d694-c0bd-458c-ae61-3f2d7cb9295a None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:06.764103 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7477faae-78a6-44b1-b260-dc459478ad5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:06.765080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7477faae-78a6-44b1-b260-dc459478ad5b None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:06.765610 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7477faae-78a6-44b1-b260-dc459478ad5b None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:06.766077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7477faae-78a6-44b1-b260-dc459478ad5b None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:06.767020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7477faae-78a6-44b1-b260-dc459478ad5b None admin] RBAC: Authorizing `identity:create_role()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:06.769287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7477faae-78a6-44b1-b260-dc459478ad5b None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:06.784849 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4584c8-d330-4dec-811b-c798e2c11f87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:06.785501 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4584c8-d330-4dec-811b-c798e2c11f87 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:06.785792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4584c8-d330-4dec-811b-c798e2c11f87 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:06.786072 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4584c8-d330-4dec-811b-c798e2c11f87 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:06.808659 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4584c8-d330-4dec-811b-c798e2c11f87 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:06.810374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4584c8-d330-4dec-811b-c798e2c11f87 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:06.835498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-7c4584c8-d330-4dec-811b-c798e2c11f87 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:43:06.835498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:06.835498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:06.835498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:06.835498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:06.835498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:06.835498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:06.835498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:06.835498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:06.835498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:06.835498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:06.835498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:06.835498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:43:06.835498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:43:06.835498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:43:06.835498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:43:06.836629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:06.836629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:06.836629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:06.836629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:06.836629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:06.836629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:06.836629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:06.836629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:06.836629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:06.836629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:06.836629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:06.836629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:06.836629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:06.836629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:06.836629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:06.836629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:06.836629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:43:06.837630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:43:06.837630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:43:06.837630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:06.837630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 13/28] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:43:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 329 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:06.844263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31237136-0d77-456d-a2a7-20b4695a893c None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:06.844564 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31237136-0d77-456d-a2a7-20b4695a893c None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:06.844851 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31237136-0d77-456d-a2a7-20b4695a893c None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:06.849353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 14/29] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:06 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:06.849903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31237136-0d77-456d-a2a7-20b4695a893c None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:06.914111 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:06.916059 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 16/30] 10.222.0.36 () {60 vars in 1131 bytes} [Mon Oct 31 14:43:06 2022] POST /identity/v3/roles => generated 219 bytes in 371 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:43:06.918488 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2415e817-b653-41c3-a25b-086cd7574ca9 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:06.919069 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2415e817-b653-41c3-a25b-086cd7574ca9 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:06.919494 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2415e817-b653-41c3-a25b-086cd7574ca9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:06.940449 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-2415e817-b653-41c3-a25b-086cd7574ca9 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:06.942285 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2415e817-b653-41c3-a25b-086cd7574ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:07.017797 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 17/31] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:07.018855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2415e817-b653-41c3-a25b-086cd7574ca9 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:07.028779 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31237136-0d77-456d-a2a7-20b4695a893c None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:07.029549 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:07.112513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7950b6f1-84b4-4b54-bbff-37fdd8eacf47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:07.112999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7950b6f1-84b4-4b54-bbff-37fdd8eacf47 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:07.113289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7950b6f1-84b4-4b54-bbff-37fdd8eacf47 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:07.113567 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7950b6f1-84b4-4b54-bbff-37fdd8eacf47 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:07.114223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7950b6f1-84b4-4b54-bbff-37fdd8eacf47 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:07.114545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7950b6f1-84b4-4b54-bbff-37fdd8eacf47 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:07.115990 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7950b6f1-84b4-4b54-bbff-37fdd8eacf47 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:07.133957 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2415e817-b653-41c3-a25b-086cd7574ca9 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:07.143662 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 15/32] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:43:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:07.152724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7950b6f1-84b4-4b54-bbff-37fdd8eacf47 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:07.184470 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d1b0a2-7009-4833-ad30-4b7af16dd7b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:07.186249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d1b0a2-7009-4833-ad30-4b7af16dd7b0 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:07.187628 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d1b0a2-7009-4833-ad30-4b7af16dd7b0 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:07.188116 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d1b0a2-7009-4833-ad30-4b7af16dd7b0 None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:07.189182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d1b0a2-7009-4833-ad30-4b7af16dd7b0 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:07.192694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d1b0a2-7009-4833-ad30-4b7af16dd7b0 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:07.219302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 18/33] 10.222.0.36 () {60 vars in 1131 bytes} [Mon Oct 31 14:43:07 2022] POST /identity/v3/roles => generated 225 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:43:07.279507 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861016be-5e5d-4d40-959f-33d7b3653445 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:07.280041 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861016be-5e5d-4d40-959f-33d7b3653445 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:07.280555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861016be-5e5d-4d40-959f-33d7b3653445 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:07.282118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 19/34] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:43:07.290086 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad6b3fc-5651-4c64-a2ca-4a91540b5fc1 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:07.290776 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad6b3fc-5651-4c64-a2ca-4a91540b5fc1 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:07.291306 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad6b3fc-5651-4c64-a2ca-4a91540b5fc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:07.297730 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7950b6f1-84b4-4b54-bbff-37fdd8eacf47 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:07.313057 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:07.333046 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-8ad6b3fc-5651-4c64-a2ca-4a91540b5fc1 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:07.334616 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8ad6b3fc-5651-4c64-a2ca-4a91540b5fc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:07.341429 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cca28c-b24f-48fc-b5a4-0be0090183f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:07.342047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cca28c-b24f-48fc-b5a4-0be0090183f9 None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:07.342340 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cca28c-b24f-48fc-b5a4-0be0090183f9 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:07.342604 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cca28c-b24f-48fc-b5a4-0be0090183f9 None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:07.343264 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cca28c-b24f-48fc-b5a4-0be0090183f9 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:07.347594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cca28c-b24f-48fc-b5a4-0be0090183f9 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:07.390773 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 16/35] 10.222.0.36 () {60 vars in 1141 bytes} [Mon Oct 31 14:43:07 2022] POST /identity/v3/projects => generated 301 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:43:07.444078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:07.446574 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 20/36] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:43:07.550233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cca28c-b24f-48fc-b5a4-0be0090183f9 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:07.688191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cca28c-b24f-48fc-b5a4-0be0090183f9 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:07.727474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74172573-5c88-4419-a826-3c3734140450 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:07.728444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74172573-5c88-4419-a826-3c3734140450 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:07.728896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74172573-5c88-4419-a826-3c3734140450 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:07.729340 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74172573-5c88-4419-a826-3c3734140450 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:07.740753 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74172573-5c88-4419-a826-3c3734140450 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:07.746420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6d1ce1-3cc0-408a-9a70-a2e04cb2ca18 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:07.746794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6d1ce1-3cc0-408a-9a70-a2e04cb2ca18 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:07.747140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6d1ce1-3cc0-408a-9a70-a2e04cb2ca18 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:07.748711 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 21/37] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:43:07.755249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5dc787-de76-4752-bbec-fd7ec0fb99bc None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:07.755533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5dc787-de76-4752-bbec-fd7ec0fb99bc None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:07.755849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5dc787-de76-4752-bbec-fd7ec0fb99bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:07.777982 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-1f5dc787-de76-4752-bbec-fd7ec0fb99bc None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:07.779306 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1f5dc787-de76-4752-bbec-fd7ec0fb99bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:07.851682 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 22/38] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:43:07.852473 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5dc787-de76-4752-bbec-fd7ec0fb99bc None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:07.857743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-74172573-5c88-4419-a826-3c3734140450 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:43:07.857743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:07.857743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:07.857743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:07.857743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:07.857743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:07.857743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:07.857743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:07.857743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:07.857743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:07.857743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:07.857743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:07.857743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:07.857743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:07.857743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:07.857743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:07.858585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:07.858585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:07.858585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:07.858585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:07.858585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:07.858585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:07.858585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:07.858585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:07.858585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:07.858585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:07.858585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:07.858585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:07.858585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:07.858585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:07.858585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:07.858585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:07.859347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:07.859347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:07.859347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:07.859347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:07.859347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:07.859347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:07.859347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:07.859347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:07.859347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:07.859347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:07.859347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:43:07.859347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:07.859347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 17/39] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:43:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:43:07.866564 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74172573-5c88-4419-a826-3c3734140450 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:07.893706 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:07.961085 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5dc787-de76-4752-bbec-fd7ec0fb99bc None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:07.974415 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74172573-5c88-4419-a826-3c3734140450 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:07.984395 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:08.001455 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7059ad48-ad34-45a2-ad10-15f559c6b20e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:08.002685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7059ad48-ad34-45a2-ad10-15f559c6b20e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:08.002894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7059ad48-ad34-45a2-ad10-15f559c6b20e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:08.003166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7059ad48-ad34-45a2-ad10-15f559c6b20e None admin] PATH_INFO: `/v3/domains/default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:08.004319 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7059ad48-ad34-45a2-ad10-15f559c6b20e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:08.005842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7059ad48-ad34-45a2-ad10-15f559c6b20e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:08.007484 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 23/40] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:43:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:08.012641 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7059ad48-ad34-45a2-ad10-15f559c6b20e None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:08.012879 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d5a659-b5bc-449e-a44d-4d738bab47e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:08.013499 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d5a659-b5bc-449e-a44d-4d738bab47e6 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:08.013790 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d5a659-b5bc-449e-a44d-4d738bab47e6 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:08.014073 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d5a659-b5bc-449e-a44d-4d738bab47e6 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:08.014752 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d5a659-b5bc-449e-a44d-4d738bab47e6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:08.015065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d5a659-b5bc-449e-a44d-4d738bab47e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:08.017072 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d5a659-b5bc-449e-a44d-4d738bab47e6 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:08.025448 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 18/41] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:08.031010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d5a659-b5bc-449e-a44d-4d738bab47e6 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:08.122757 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:08.122920 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7059ad48-ad34-45a2-ad10-15f559c6b20e None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:08.137848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d5a659-b5bc-449e-a44d-4d738bab47e6 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:08.158359 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa78ba0c-f624-4014-a2d9-03b240a1cbb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:08.158979 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa78ba0c-f624-4014-a2d9-03b240a1cbb5 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:08.159258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa78ba0c-f624-4014-a2d9-03b240a1cbb5 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:08.159536 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa78ba0c-f624-4014-a2d9-03b240a1cbb5 None admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:08.160217 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa78ba0c-f624-4014-a2d9-03b240a1cbb5 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:08.161667 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa78ba0c-f624-4014-a2d9-03b240a1cbb5 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:08.172158 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b83d6b-03d3-4add-8f07-2310709cc098 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:08.172785 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b83d6b-03d3-4add-8f07-2310709cc098 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:08.173067 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b83d6b-03d3-4add-8f07-2310709cc098 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:08.173448 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b83d6b-03d3-4add-8f07-2310709cc098 None admin] PATH_INFO: `/v3/users/7c12195059ec40bfafbb7be8e9f46734` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:08.174670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b83d6b-03d3-4add-8f07-2310709cc098 None admin] RBAC: Authorizing `identity:get_user(user_id=7c12195059ec40bfafbb7be8e9f46734)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:08.176144 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b83d6b-03d3-4add-8f07-2310709cc098 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:08.177758 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 19/42] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:08 2022] GET /identity/v3/users/7c12195059ec40bfafbb7be8e9f46734 => generated 247 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:08.178505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b83d6b-03d3-4add-8f07-2310709cc098 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:08.190425 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 24/43] 10.222.0.36 () {60 vars in 1141 bytes} [Mon Oct 31 14:43:08 2022] POST /identity/v3/projects => generated 312 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:43:08.191510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa78ba0c-f624-4014-a2d9-03b240a1cbb5 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:08.252829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:08.289780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b83d6b-03d3-4add-8f07-2310709cc098 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:08.325082 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa78ba0c-f624-4014-a2d9-03b240a1cbb5 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:08.326636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28241df-06a4-4997-8f47-f4ec0c4a62e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:08.327295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28241df-06a4-4997-8f47-f4ec0c4a62e1 None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:08.327599 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28241df-06a4-4997-8f47-f4ec0c4a62e1 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:08.327873 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28241df-06a4-4997-8f47-f4ec0c4a62e1 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:08.328583 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28241df-06a4-4997-8f47-f4ec0c4a62e1 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:08.330779 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28241df-06a4-4997-8f47-f4ec0c4a62e1 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:08.348145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 20/44] 10.222.0.36 () {60 vars in 1131 bytes} [Mon Oct 31 14:43:08 2022] POST /identity/v3/roles => generated 223 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:43:08.374384 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10289e35-baec-4bb1-955a-b66146c9887d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:08.375055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10289e35-baec-4bb1-955a-b66146c9887d None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:08.375337 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10289e35-baec-4bb1-955a-b66146c9887d None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:08.376865 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10289e35-baec-4bb1-955a-b66146c9887d None admin] PATH_INFO: `/v3/domains/default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:08.377918 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10289e35-baec-4bb1-955a-b66146c9887d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:08.379353 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10289e35-baec-4bb1-955a-b66146c9887d None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:08.382099 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 25/45] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:43:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:08.387094 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28241df-06a4-4997-8f47-f4ec0c4a62e1 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:08.405839 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:08.505846 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28241df-06a4-4997-8f47-f4ec0c4a62e1 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:08.547937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e76384-a5af-4af6-b985-28176a092fd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:08.549328 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e76384-a5af-4af6-b985-28176a092fd7 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:08.549624 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e76384-a5af-4af6-b985-28176a092fd7 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:08.549906 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e76384-a5af-4af6-b985-28176a092fd7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:08.550658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e76384-a5af-4af6-b985-28176a092fd7 None admin] RBAC: Adding query filter params (role.id=57572a37675d4c418e13794a7b8f3a5d, scope.domain.id=default, user.id=7c12195059ec40bfafbb7be8e9f46734) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:08.550977 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e76384-a5af-4af6-b985-28176a092fd7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:08.552519 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e76384-a5af-4af6-b985-28176a092fd7 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:08.560375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 21/46] 10.222.0.36 () {56 vars in 1322 bytes} [Mon Oct 31 14:43:08 2022] GET /identity/v3/role_assignments?user.id=7c12195059ec40bfafbb7be8e9f46734&scope.domain.id=default&role.id=57572a37675d4c418e13794a7b8f3a5d => generated 530 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:10.412644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f373a173-ce07-4b78-9394-6076c9b4de00 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:10.413163 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f373a173-ce07-4b78-9394-6076c9b4de00 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:10.413245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f373a173-ce07-4b78-9394-6076c9b4de00 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:10.414535 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 26/47] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:10.419844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c49171-ba85-4266-abcc-f083f2ec4e58 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:10.420230 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c49171-ba85-4266-abcc-f083f2ec4e58 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:10.420402 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c49171-ba85-4266-abcc-f083f2ec4e58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:10.438133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-29c49171-ba85-4266-abcc-f083f2ec4e58 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:10.439543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-29c49171-ba85-4266-abcc-f083f2ec4e58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:10.504725 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 22/48] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:10.773851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f373a173-ce07-4b78-9394-6076c9b4de00 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:10.871606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f373a173-ce07-4b78-9394-6076c9b4de00 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:10.880669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:10.905519 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafb6d92-e9ab-4cd9-94a2-dae98861e203 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:10.906134 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafb6d92-e9ab-4cd9-94a2-dae98861e203 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:10.906415 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafb6d92-e9ab-4cd9-94a2-dae98861e203 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:10.906711 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafb6d92-e9ab-4cd9-94a2-dae98861e203 None admin] PATH_INFO: `/v3/domains/default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:10.907839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafb6d92-e9ab-4cd9-94a2-dae98861e203 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:10.909266 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafb6d92-e9ab-4cd9-94a2-dae98861e203 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:10.910805 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 27/49] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:43:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:10.916404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c49171-ba85-4266-abcc-f083f2ec4e58 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:10.926352 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:11.014898 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c49171-ba85-4266-abcc-f083f2ec4e58 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:11.048905 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645398a6-1c0e-4e66-9be2-e9f03f4e825b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:11.049601 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645398a6-1c0e-4e66-9be2-e9f03f4e825b None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:11.049884 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645398a6-1c0e-4e66-9be2-e9f03f4e825b None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:11.050161 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645398a6-1c0e-4e66-9be2-e9f03f4e825b None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:11.050818 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645398a6-1c0e-4e66-9be2-e9f03f4e825b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:11.052291 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645398a6-1c0e-4e66-9be2-e9f03f4e825b None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:11.078903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 23/50] 10.222.0.36 () {60 vars in 1140 bytes} [Mon Oct 31 14:43:10 2022] POST /identity/v3/projects => generated 298 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:43:13.223102 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73880227-ca37-44d0-b584-5cf98776859c None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:13.223706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73880227-ca37-44d0-b584-5cf98776859c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:13.223706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73880227-ca37-44d0-b584-5cf98776859c None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:13.225028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 28/51] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:13.230328 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4557ac-c423-4d44-aa1d-575dbeb9d295 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:13.230704 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4557ac-c423-4d44-aa1d-575dbeb9d295 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:13.230879 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4557ac-c423-4d44-aa1d-575dbeb9d295 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:13.249913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-fc4557ac-c423-4d44-aa1d-575dbeb9d295 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:13.251781 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-fc4557ac-c423-4d44-aa1d-575dbeb9d295 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:13.306911 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:13.319653 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 24/52] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:13.588274 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73880227-ca37-44d0-b584-5cf98776859c None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:13.688138 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73880227-ca37-44d0-b584-5cf98776859c None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:13.720760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4dbc49-df20-49b1-a99e-aac9752ca1a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:13.721365 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4dbc49-df20-49b1-a99e-aac9752ca1a9 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:13.721644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4dbc49-df20-49b1-a99e-aac9752ca1a9 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:13.722168 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4dbc49-df20-49b1-a99e-aac9752ca1a9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:13.723257 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4dbc49-df20-49b1-a99e-aac9752ca1a9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:13.724708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4dbc49-df20-49b1-a99e-aac9752ca1a9 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:13.726238 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 29/53] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:43:13 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:13.731745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4557ac-c423-4d44-aa1d-575dbeb9d295 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:13.827964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4557ac-c423-4d44-aa1d-575dbeb9d295 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:13.862226 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a45e167-88b8-4181-a498-e21ea734d318 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:13.862822 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a45e167-88b8-4181-a498-e21ea734d318 None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:13.863101 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a45e167-88b8-4181-a498-e21ea734d318 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:13.863374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a45e167-88b8-4181-a498-e21ea734d318 None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:13.864050 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a45e167-88b8-4181-a498-e21ea734d318 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:13.865501 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a45e167-88b8-4181-a498-e21ea734d318 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:13.893731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 25/54] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:43:13 2022] POST /identity/v3/users => generated 275 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:43:17.051584 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98886166-a4b8-42fa-a06b-2e7bf883fb4d None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:17.051584 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98886166-a4b8-42fa-a06b-2e7bf883fb4d None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:17.051584 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98886166-a4b8-42fa-a06b-2e7bf883fb4d None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:17.052674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 30/55] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:17.057670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d88b083-249c-479c-93ee-657b4a1698b7 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:17.058102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d88b083-249c-479c-93ee-657b4a1698b7 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:17.058401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d88b083-249c-479c-93ee-657b4a1698b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:17.081754 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-3d88b083-249c-479c-93ee-657b4a1698b7 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:17.082958 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-3d88b083-249c-479c-93ee-657b4a1698b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:17.142892 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:17.176861 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 26/56] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:17.253763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3b3a4b-20ac-430d-88ee-310b4c6df033 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:17.254398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3b3a4b-20ac-430d-88ee-310b4c6df033 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:17.254877 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3b3a4b-20ac-430d-88ee-310b4c6df033 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:17.256707 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 31/57] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:17.261764 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078c71c5-6501-42d9-95f1-ceba3e545f44 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:17.262054 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078c71c5-6501-42d9-95f1-ceba3e545f44 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:17.262335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078c71c5-6501-42d9-95f1-ceba3e545f44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:17.292853 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-078c71c5-6501-42d9-95f1-ceba3e545f44 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:17.294410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-078c71c5-6501-42d9-95f1-ceba3e545f44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:17.386290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 27/58] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:17.499939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0f2673-4f54-42ed-8b48-d10aa3ba48bb None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:17.499939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0f2673-4f54-42ed-8b48-d10aa3ba48bb None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:17.499939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0f2673-4f54-42ed-8b48-d10aa3ba48bb None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:17.500646 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 32/59] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:17.505590 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a186032-b51b-43b0-b0b7-8fb4616972a0 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:17.505890 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a186032-b51b-43b0-b0b7-8fb4616972a0 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:17.506175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a186032-b51b-43b0-b0b7-8fb4616972a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:17.509588 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0f2673-4f54-42ed-8b48-d10aa3ba48bb None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:17.541606 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-0a186032-b51b-43b0-b0b7-8fb4616972a0 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:17.542828 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-0a186032-b51b-43b0-b0b7-8fb4616972a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:17.606457 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:17.692305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 28/60] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:17.708903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0f2673-4f54-42ed-8b48-d10aa3ba48bb None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:17.734084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a186032-b51b-43b0-b0b7-8fb4616972a0 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:17.752569 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b99754-16a6-4e24-95e7-594d937369a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:17.753796 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b99754-16a6-4e24-95e7-594d937369a5 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:17.754660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b99754-16a6-4e24-95e7-594d937369a5 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:17.755426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b99754-16a6-4e24-95e7-594d937369a5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:17.771276 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b99754-16a6-4e24-95e7-594d937369a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:17.777049 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:17.779424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-13b99754-16a6-4e24-95e7-594d937369a5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:17.779424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:17.779424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:17.779424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:17.779424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:17.779424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:17.779424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:17.779424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:17.779424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:17.779424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:17.779424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:17.779424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:17.779424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:17.779424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:17.779424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:17.779424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:17.780750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:17.780750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:17.780750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:17.780750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:17.780750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:17.780750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:17.780750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:17.780750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:17.780750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:17.780750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:17.780750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:17.780750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:17.780750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:17.780750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:17.780750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:17.780750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:17.781728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:17.781728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:17.781728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:17.781728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:17.781728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:17.781728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:17.781728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:17.781728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:17.781728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:17.781728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:17.781728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:17.781728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:17.783466 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 33/61] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:17.784865 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd0f33d-2969-4b62-b544-10d0c214554b None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:17.785243 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd0f33d-2969-4b62-b544-10d0c214554b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:17.785610 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd0f33d-2969-4b62-b544-10d0c214554b None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:17.787685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 34/62] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:17.793688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd0f33d-2969-4b62-b544-10d0c214554b None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:17.872537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a186032-b51b-43b0-b0b7-8fb4616972a0 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:17.921105 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10e91a0-f155-48b3-8ff2-df7c5574ad04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:17.922298 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10e91a0-f155-48b3-8ff2-df7c5574ad04 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:17.922637 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10e91a0-f155-48b3-8ff2-df7c5574ad04 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:17.922950 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10e91a0-f155-48b3-8ff2-df7c5574ad04 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:17.930017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd0f33d-2969-4b62-b544-10d0c214554b None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:17.935783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:17.942147 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10e91a0-f155-48b3-8ff2-df7c5574ad04 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:17.947329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-a10e91a0-f155-48b3-8ff2-df7c5574ad04 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:43:17.947329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:17.947329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:17.947329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:17.947329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:17.947329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:17.947329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:17.947329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:17.947329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:17.947329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:17.947329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:17.947329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:17.947329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:17.947329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:17.947329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:17.947329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:17.948411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:17.948411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:17.948411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:17.948411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:17.948411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:17.948411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:17.948411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:17.948411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:17.948411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:17.948411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:17.948411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:17.948411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:17.948411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:17.948411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:17.948411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:17.948411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:17.949175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:17.949175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:17.949175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:17.949175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:17.949175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:17.949175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:17.949175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:17.949175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:17.949175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:17.949175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:17.949175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:43:17.949175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:17.953794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 29/63] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:43:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:43:17.956401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c79405b-3aa3-4d15-b7c5-2498aacbfbda None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:17.956589 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c79405b-3aa3-4d15-b7c5-2498aacbfbda None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:17.956857 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c79405b-3aa3-4d15-b7c5-2498aacbfbda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:17.975904 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-4c79405b-3aa3-4d15-b7c5-2498aacbfbda None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:17.977097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-4c79405b-3aa3-4d15-b7c5-2498aacbfbda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:17.979323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca75d78-b737-46cc-a37a-ca71838f239f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:17.980299 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca75d78-b737-46cc-a37a-ca71838f239f None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:17.980784 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca75d78-b737-46cc-a37a-ca71838f239f None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:17.981275 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca75d78-b737-46cc-a37a-ca71838f239f None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:17.982145 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca75d78-b737-46cc-a37a-ca71838f239f None admin] RBAC: Adding query filter params (name=member) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:17.982651 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca75d78-b737-46cc-a37a-ca71838f239f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:17.984430 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca75d78-b737-46cc-a37a-ca71838f239f None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:17.997474 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 35/64] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:17 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:17.998596 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca75d78-b737-46cc-a37a-ca71838f239f None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:18.050965 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 30/65] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:18.051843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c79405b-3aa3-4d15-b7c5-2498aacbfbda None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:18.078244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:18.106442 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca75d78-b737-46cc-a37a-ca71838f239f None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:18.143821 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b84b88-6ee4-49db-8e0c-f0bf1198a7ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:18.144444 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b84b88-6ee4-49db-8e0c-f0bf1198a7ba None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:18.144727 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b84b88-6ee4-49db-8e0c-f0bf1198a7ba None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:18.145003 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b84b88-6ee4-49db-8e0c-f0bf1198a7ba None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:18.145669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b84b88-6ee4-49db-8e0c-f0bf1198a7ba None admin] RBAC: Adding query filter params (name=admin) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:18.145978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b84b88-6ee4-49db-8e0c-f0bf1198a7ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:18.147355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b84b88-6ee4-49db-8e0c-f0bf1198a7ba None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:18.161585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 36/66] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:18.161585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b84b88-6ee4-49db-8e0c-f0bf1198a7ba None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:18.171837 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c79405b-3aa3-4d15-b7c5-2498aacbfbda None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:18.213124 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d9291e-07dd-4105-b905-ae7aaf612c1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:18.213701 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d9291e-07dd-4105-b905-ae7aaf612c1b None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:18.213990 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d9291e-07dd-4105-b905-ae7aaf612c1b None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:18.214271 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d9291e-07dd-4105-b905-ae7aaf612c1b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:18.227339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d9291e-07dd-4105-b905-ae7aaf612c1b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:18.232434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-c0d9291e-07dd-4105-b905-ae7aaf612c1b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:43:18.232434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:18.232434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:18.232434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:18.232434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:18.232434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:18.232434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:18.232434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:18.232434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:18.232434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:18.232434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:18.232434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:18.232434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:18.232434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:18.232434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:18.232434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:18.233316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:18.233316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:18.233316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:18.233316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:18.233316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:18.233316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:18.233316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:18.233316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:18.233316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:18.233316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:18.233316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:18.233316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:18.233316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:18.233316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:18.233316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:18.233316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:18.234063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:18.234063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:18.234063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:18.234063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:18.234063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:18.234063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:18.234063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:18.234063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:18.234063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:18.234063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:18.234063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:43:18.234063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:18.235897 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 31/67] 10.222.0.36 () {56 vars in 1114 bytes} [Mon Oct 31 14:43:18 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:18.237534 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52836f75-baca-472d-aeba-015f330934c3 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:18.238050 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52836f75-baca-472d-aeba-015f330934c3 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:18.238534 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52836f75-baca-472d-aeba-015f330934c3 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:18.240766 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 32/68] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:18.241776 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52836f75-baca-472d-aeba-015f330934c3 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:18.301246 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:18.347919 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:18.364174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52836f75-baca-472d-aeba-015f330934c3 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:18.375135 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b84b88-6ee4-49db-8e0c-f0bf1198a7ba None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:18.398672 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfd444c-3348-4486-8b38-89ff6f078ad4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:18.399300 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfd444c-3348-4486-8b38-89ff6f078ad4 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:18.399615 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfd444c-3348-4486-8b38-89ff6f078ad4 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:18.399894 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfd444c-3348-4486-8b38-89ff6f078ad4 None admin] PATH_INFO: `/v3/users/7c12195059ec40bfafbb7be8e9f46734` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:18.401035 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfd444c-3348-4486-8b38-89ff6f078ad4 None admin] RBAC: Authorizing `identity:get_user(user_id=7c12195059ec40bfafbb7be8e9f46734)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:18.402519 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfd444c-3348-4486-8b38-89ff6f078ad4 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:18.404169 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 33/69] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:18 2022] GET /identity/v3/users/7c12195059ec40bfafbb7be8e9f46734 => generated 247 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:18.404914 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfd444c-3348-4486-8b38-89ff6f078ad4 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:18.410628 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c541c23-493d-487b-bcde-a227f23144c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:18.411340 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c541c23-493d-487b-bcde-a227f23144c8 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:18.411698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c541c23-493d-487b-bcde-a227f23144c8 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:18.411986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c541c23-493d-487b-bcde-a227f23144c8 None admin] PATH_INFO: `/v3/users/b0dd86f9993647219d6b755d3d7c09f5` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:18.427120 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c541c23-493d-487b-bcde-a227f23144c8 None admin] RBAC: Authorizing `identity:get_user(user_id=b0dd86f9993647219d6b755d3d7c09f5)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:18.428626 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c541c23-493d-487b-bcde-a227f23144c8 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:18.430402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 37/70] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:18 2022] GET /identity/v3/users/b0dd86f9993647219d6b755d3d7c09f5 => generated 275 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:18.431690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7854d226-e604-4a5f-9f1c-fa5d60e27ed8 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:18.431975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7854d226-e604-4a5f-9f1c-fa5d60e27ed8 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:18.432251 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7854d226-e604-4a5f-9f1c-fa5d60e27ed8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:18.449842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-7854d226-e604-4a5f-9f1c-fa5d60e27ed8 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:18.451037 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7854d226-e604-4a5f-9f1c-fa5d60e27ed8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:18.509697 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfd444c-3348-4486-8b38-89ff6f078ad4 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:18.522854 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 38/71] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:18.534316 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7854d226-e604-4a5f-9f1c-fa5d60e27ed8 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:18.542544 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:18.557830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df854228-9f0a-47e2-bdd3-03456e0d7c9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:18.558517 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df854228-9f0a-47e2-bdd3-03456e0d7c9e None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:18.558973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df854228-9f0a-47e2-bdd3-03456e0d7c9e None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:18.559153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df854228-9f0a-47e2-bdd3-03456e0d7c9e None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:18.559834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df854228-9f0a-47e2-bdd3-03456e0d7c9e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:18.560139 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df854228-9f0a-47e2-bdd3-03456e0d7c9e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:18.561514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df854228-9f0a-47e2-bdd3-03456e0d7c9e None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:18.574911 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 34/72] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:43:18 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:18.574911 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df854228-9f0a-47e2-bdd3-03456e0d7c9e None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:18.591681 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:18.638972 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7854d226-e604-4a5f-9f1c-fa5d60e27ed8 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:18.677588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df854228-9f0a-47e2-bdd3-03456e0d7c9e None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:18.677906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79169f50-57e0-4788-bad9-c63504909491 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:18.678736 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79169f50-57e0-4788-bad9-c63504909491 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:18.679214 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79169f50-57e0-4788-bad9-c63504909491 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:18.679710 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79169f50-57e0-4788-bad9-c63504909491 None admin] PATH_INFO: `/v3/roles/member` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:18.691774 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79169f50-57e0-4788-bad9-c63504909491 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:18.697249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-79169f50-57e0-4788-bad9-c63504909491 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:18.697249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:18.697249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:18.697249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:18.697249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:18.697249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:18.697249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:18.697249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:18.697249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:18.697249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:18.697249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:18.697249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:18.697249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:18.697249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:18.697249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:18.697249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:18.698087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:18.698087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:18.698087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:18.698087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:18.698087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:18.698087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:18.698087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:18.698087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:18.698087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:18.698087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:18.698087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:18.698087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:18.698087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:18.698087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:18.698087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:18.698087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:18.698851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:18.698851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:18.698851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:18.698851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:18.698851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:18.698851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:18.698851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:18.698851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:18.698851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:18.698851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:18.698851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:18.698851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:18.701248 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 39/73] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:18.702934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79169f50-57e0-4788-bad9-c63504909491 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:18.716070 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04039e7-53fa-4627-9d46-b59946d41512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:18.716698 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04039e7-53fa-4627-9d46-b59946d41512 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:18.716978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04039e7-53fa-4627-9d46-b59946d41512 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:18.717257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04039e7-53fa-4627-9d46-b59946d41512 None admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:18.718393 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04039e7-53fa-4627-9d46-b59946d41512 None admin] RBAC: Authorizing `identity:get_project(project_id=dc35ec7707ee4a09b3784f8b3e76686c)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:18.720414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04039e7-53fa-4627-9d46-b59946d41512 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:18.722499 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 35/74] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:18 2022] GET /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c => generated 298 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:18.723256 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04039e7-53fa-4627-9d46-b59946d41512 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:18.796622 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:18.811118 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:18.820275 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79169f50-57e0-4788-bad9-c63504909491 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:18.827058 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04039e7-53fa-4627-9d46-b59946d41512 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:18.855380 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21afe7e-5935-4828-80b5-43eca8056a21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:18.856048 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21afe7e-5935-4828-80b5-43eca8056a21 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:18.856333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21afe7e-5935-4828-80b5-43eca8056a21 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:18.856611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21afe7e-5935-4828-80b5-43eca8056a21 None admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:18.857755 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21afe7e-5935-4828-80b5-43eca8056a21 None admin] RBAC: Authorizing `identity:get_project(project_id=dc35ec7707ee4a09b3784f8b3e76686c)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:18.859197 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21afe7e-5935-4828-80b5-43eca8056a21 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:18.865749 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 40/75] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:18 2022] GET /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:18.865899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c576e5-062c-4877-963d-29dbf1aeaee1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:18.866002 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21afe7e-5935-4828-80b5-43eca8056a21 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:18.866095 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c576e5-062c-4877-963d-29dbf1aeaee1 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:18.866095 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c576e5-062c-4877-963d-29dbf1aeaee1 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:18.866095 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c576e5-062c-4877-963d-29dbf1aeaee1 None admin] PATH_INFO: `/v3/users/b0dd86f9993647219d6b755d3d7c09f5` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:18.866095 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c576e5-062c-4877-963d-29dbf1aeaee1 None admin] RBAC: Authorizing `identity:get_user(user_id=b0dd86f9993647219d6b755d3d7c09f5)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:18.866095 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c576e5-062c-4877-963d-29dbf1aeaee1 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:18.867118 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 36/76] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:18 2022] GET /identity/v3/users/b0dd86f9993647219d6b755d3d7c09f5 => generated 275 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:18.867885 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c576e5-062c-4877-963d-29dbf1aeaee1 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:18.963740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21afe7e-5935-4828-80b5-43eca8056a21 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:18.969281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c576e5-062c-4877-963d-29dbf1aeaee1 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:18.998644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986b116e-6f32-43f6-a986-6f446f840e97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:18.999262 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986b116e-6f32-43f6-a986-6f446f840e97 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:18.999541 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986b116e-6f32-43f6-a986-6f446f840e97 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:18.999847 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986b116e-6f32-43f6-a986-6f446f840e97 None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:19.000505 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986b116e-6f32-43f6-a986-6f446f840e97 None admin] RBAC: Adding query filter params (name=member) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:19.001333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986b116e-6f32-43f6-a986-6f446f840e97 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:19.002719 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986b116e-6f32-43f6-a986-6f446f840e97 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:19.002812 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97638c56-6a9a-4f3e-915b-0f19ae2eaf20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:19.003373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97638c56-6a9a-4f3e-915b-0f19ae2eaf20 None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:19.003753 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97638c56-6a9a-4f3e-915b-0f19ae2eaf20 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:19.004098 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97638c56-6a9a-4f3e-915b-0f19ae2eaf20 None admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c/users/7c12195059ec40bfafbb7be8e9f46734/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:19.006041 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97638c56-6a9a-4f3e-915b-0f19ae2eaf20 None admin] RBAC: Authorizing `identity:create_grant(project_id=dc35ec7707ee4a09b3784f8b3e76686c, user_id=7c12195059ec40bfafbb7be8e9f46734, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:19.007469 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97638c56-6a9a-4f3e-915b-0f19ae2eaf20 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:19.010599 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 41/77] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:19.011463 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986b116e-6f32-43f6-a986-6f446f840e97 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:19.021424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 37/78] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:43:18 2022] PUT /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c/users/7c12195059ec40bfafbb7be8e9f46734/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:43:19.022050 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97638c56-6a9a-4f3e-915b-0f19ae2eaf20 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:19.023283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:19.038086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:19.129636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986b116e-6f32-43f6-a986-6f446f840e97 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:19.144183 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97638c56-6a9a-4f3e-915b-0f19ae2eaf20 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:19.168095 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c252257-c29d-41e8-8d95-2eb6b1bc8d83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:19.168783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c252257-c29d-41e8-8d95-2eb6b1bc8d83 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:19.169056 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c252257-c29d-41e8-8d95-2eb6b1bc8d83 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:19.169307 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c252257-c29d-41e8-8d95-2eb6b1bc8d83 None admin] PATH_INFO: `/v3/domains/default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:19.170519 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c252257-c29d-41e8-8d95-2eb6b1bc8d83 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:19.172041 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c252257-c29d-41e8-8d95-2eb6b1bc8d83 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:19.174053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 42/79] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:43:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:19.174709 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c252257-c29d-41e8-8d95-2eb6b1bc8d83 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:19.183962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6252b5-62d8-46d8-b971-53703b6345fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:19.184690 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6252b5-62d8-46d8-b971-53703b6345fe None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:19.184928 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6252b5-62d8-46d8-b971-53703b6345fe None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:19.185210 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6252b5-62d8-46d8-b971-53703b6345fe None admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c/users/b0dd86f9993647219d6b755d3d7c09f5/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:19.187195 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6252b5-62d8-46d8-b971-53703b6345fe None admin] RBAC: Authorizing `identity:create_grant(project_id=dc35ec7707ee4a09b3784f8b3e76686c, user_id=b0dd86f9993647219d6b755d3d7c09f5, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:19.189284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6252b5-62d8-46d8-b971-53703b6345fe None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:19.205842 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 38/80] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:43:19 2022] PUT /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c/users/b0dd86f9993647219d6b755d3d7c09f5/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:43:19.206573 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6252b5-62d8-46d8-b971-53703b6345fe None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:19.281840 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:19.314420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:19.321279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c252257-c29d-41e8-8d95-2eb6b1bc8d83 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:19.363734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6252b5-62d8-46d8-b971-53703b6345fe None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:19.382420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5b0b76-6c57-438e-ac6f-6f1fe8e181ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:19.383766 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5b0b76-6c57-438e-ac6f-6f1fe8e181ac None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:19.384714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5b0b76-6c57-438e-ac6f-6f1fe8e181ac None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:19.385254 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5b0b76-6c57-438e-ac6f-6f1fe8e181ac None admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:19.386673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5b0b76-6c57-438e-ac6f-6f1fe8e181ac None admin] RBAC: Authorizing `identity:get_project(project_id=dc35ec7707ee4a09b3784f8b3e76686c)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:19.388830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5b0b76-6c57-438e-ac6f-6f1fe8e181ac None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:19.391142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 43/81] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:19 2022] GET /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c => generated 298 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:19.392501 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5b0b76-6c57-438e-ac6f-6f1fe8e181ac None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:19.405757 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4096a169-cd21-4445-980f-f79a16b37e1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:19.406401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4096a169-cd21-4445-980f-f79a16b37e1a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:19.406684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4096a169-cd21-4445-980f-f79a16b37e1a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:19.407024 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4096a169-cd21-4445-980f-f79a16b37e1a None admin] PATH_INFO: `/v3/users/b0dd86f9993647219d6b755d3d7c09f5` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:19.408724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4096a169-cd21-4445-980f-f79a16b37e1a None admin] RBAC: Authorizing `identity:get_user(user_id=b0dd86f9993647219d6b755d3d7c09f5)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:19.410175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4096a169-cd21-4445-980f-f79a16b37e1a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:19.411820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 39/82] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:19 2022] GET /identity/v3/users/b0dd86f9993647219d6b755d3d7c09f5 => generated 275 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:19.412561 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4096a169-cd21-4445-980f-f79a16b37e1a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:19.517396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5b0b76-6c57-438e-ac6f-6f1fe8e181ac None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:19.540057 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4096a169-cd21-4445-980f-f79a16b37e1a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:19.559039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6135d2-6c69-4b3a-b1bd-17f92360b4b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:19.560060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6135d2-6c69-4b3a-b1bd-17f92360b4b6 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:19.560547 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6135d2-6c69-4b3a-b1bd-17f92360b4b6 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:19.561023 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6135d2-6c69-4b3a-b1bd-17f92360b4b6 None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:19.561880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6135d2-6c69-4b3a-b1bd-17f92360b4b6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:19.563652 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6135d2-6c69-4b3a-b1bd-17f92360b4b6 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:19.582249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2315d7c7-1fd0-4c01-94e4-6c21f885dbdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:19.582920 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2315d7c7-1fd0-4c01-94e4-6c21f885dbdd None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:19.583203 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2315d7c7-1fd0-4c01-94e4-6c21f885dbdd None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:19.583482 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2315d7c7-1fd0-4c01-94e4-6c21f885dbdd None admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c/users/b0dd86f9993647219d6b755d3d7c09f5/roles/addf152a48404c27b7d71386e81b8f76` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:19.586105 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2315d7c7-1fd0-4c01-94e4-6c21f885dbdd None admin] RBAC: Authorizing `identity:create_grant(project_id=dc35ec7707ee4a09b3784f8b3e76686c, user_id=b0dd86f9993647219d6b755d3d7c09f5, role_id=addf152a48404c27b7d71386e81b8f76)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:19.587621 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2315d7c7-1fd0-4c01-94e4-6c21f885dbdd None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:19.589667 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:19.600574 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:19.603800 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 44/83] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:43:19 2022] POST /identity/v3/users => generated 289 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:43:19.605239 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6135d2-6c69-4b3a-b1bd-17f92360b4b6 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:19.610115 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 40/84] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:43:19 2022] PUT /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c/users/b0dd86f9993647219d6b755d3d7c09f5/roles/addf152a48404c27b7d71386e81b8f76 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:43:19.779030 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6135d2-6c69-4b3a-b1bd-17f92360b4b6 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:19.824577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7abb854-9e9d-4a21-a08e-68316801be09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:19.825240 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7abb854-9e9d-4a21-a08e-68316801be09 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:19.825530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7abb854-9e9d-4a21-a08e-68316801be09 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:19.825796 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7abb854-9e9d-4a21-a08e-68316801be09 None admin] PATH_INFO: `/v3/projects/5327d62517ed4878979f28cc0cd2e1bb` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:19.827032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7abb854-9e9d-4a21-a08e-68316801be09 None admin] RBAC: Authorizing `identity:get_project(project_id=5327d62517ed4878979f28cc0cd2e1bb)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:19.828691 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7abb854-9e9d-4a21-a08e-68316801be09 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:19.831047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 45/85] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:19 2022] GET /identity/v3/projects/5327d62517ed4878979f28cc0cd2e1bb => generated 312 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:19.837042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2315d7c7-1fd0-4c01-94e4-6c21f885dbdd None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:19.952792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2315d7c7-1fd0-4c01-94e4-6c21f885dbdd None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:19.992848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da249e8-6b29-47e1-9ad1-f90a31671cde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:19.993576 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da249e8-6b29-47e1-9ad1-f90a31671cde None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:19.993859 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da249e8-6b29-47e1-9ad1-f90a31671cde None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:19.994135 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da249e8-6b29-47e1-9ad1-f90a31671cde None admin] PATH_INFO: `/v3/projects/5327d62517ed4878979f28cc0cd2e1bb/users/b0dd86f9993647219d6b755d3d7c09f5/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:19.996179 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da249e8-6b29-47e1-9ad1-f90a31671cde None admin] RBAC: Authorizing `identity:create_grant(project_id=5327d62517ed4878979f28cc0cd2e1bb, user_id=b0dd86f9993647219d6b755d3d7c09f5, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:19.997638 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da249e8-6b29-47e1-9ad1-f90a31671cde None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:20.011262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 41/86] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:43:19 2022] PUT /identity/v3/projects/5327d62517ed4878979f28cc0cd2e1bb/users/b0dd86f9993647219d6b755d3d7c09f5/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:43:21.377402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a358bf2c-2ae9-47a8-ba8a-8470cea9dc50 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:21.377915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a358bf2c-2ae9-47a8-ba8a-8470cea9dc50 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:21.377915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a358bf2c-2ae9-47a8-ba8a-8470cea9dc50 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:21.379286 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 46/87] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:21.387696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e184fd08-0d6f-488b-a43b-db42d854da2a None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:21.388159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e184fd08-0d6f-488b-a43b-db42d854da2a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:21.388236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e184fd08-0d6f-488b-a43b-db42d854da2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:21.415528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-e184fd08-0d6f-488b-a43b-db42d854da2a None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:21.416811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e184fd08-0d6f-488b-a43b-db42d854da2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:21.482182 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:21.496713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 42/88] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:43:21.791371 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a358bf2c-2ae9-47a8-ba8a-8470cea9dc50 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:21.869303 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:21.909180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a358bf2c-2ae9-47a8-ba8a-8470cea9dc50 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:21.951027 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9ce8e6-93a0-4b4c-9451-581c56dd12cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:21.951724 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9ce8e6-93a0-4b4c-9451-581c56dd12cd None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:21.952009 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9ce8e6-93a0-4b4c-9451-581c56dd12cd None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:21.952288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9ce8e6-93a0-4b4c-9451-581c56dd12cd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:21.970373 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9ce8e6-93a0-4b4c-9451-581c56dd12cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:21.975611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-cd9ce8e6-93a0-4b4c-9451-581c56dd12cd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:43:21.975611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:21.975611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:21.975611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:21.975611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:21.975611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:21.975611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:21.975611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:21.975611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:21.975611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:21.975611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:21.975611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:21.975611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:21.975611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:21.975611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:21.975611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:21.976459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:21.976459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:21.976459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:21.976459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:21.976459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:21.976459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:21.976459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:21.976459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:21.976459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:21.976459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:21.976459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:21.976459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:21.976459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:21.976459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:21.976459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:21.976459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:21.977231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:21.977231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:21.977231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:21.977231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:21.977231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:21.977231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:21.977231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:21.977231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:21.977231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:21.977231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:21.977231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:43:21.977231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:21.977231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 47/89] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:43:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:21.987665 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e184fd08-0d6f-488b-a43b-db42d854da2a None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:22.092370 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100b000e-9a8c-48a4-8ebf-368551bf3959 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:22.092679 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100b000e-9a8c-48a4-8ebf-368551bf3959 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:22.092874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100b000e-9a8c-48a4-8ebf-368551bf3959 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:22.094920 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 48/90] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:22.101590 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e2fc9d-b97d-41de-a137-638691acd220 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:22.102186 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e2fc9d-b97d-41de-a137-638691acd220 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:22.102731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e2fc9d-b97d-41de-a137-638691acd220 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:22.122826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-a5e2fc9d-b97d-41de-a137-638691acd220 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:22.124526 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a5e2fc9d-b97d-41de-a137-638691acd220 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:22.171967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e184fd08-0d6f-488b-a43b-db42d854da2a None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:22.229647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3ddfc6-aaf2-4f1f-9895-f42779e2b418 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:22.230621 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3ddfc6-aaf2-4f1f-9895-f42779e2b418 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:22.231134 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3ddfc6-aaf2-4f1f-9895-f42779e2b418 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:22.231636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3ddfc6-aaf2-4f1f-9895-f42779e2b418 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:22.232624 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3ddfc6-aaf2-4f1f-9895-f42779e2b418 None admin] RBAC: Adding query filter params (name=admin) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:22.233391 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3ddfc6-aaf2-4f1f-9895-f42779e2b418 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:22.235080 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3ddfc6-aaf2-4f1f-9895-f42779e2b418 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:22.244947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 43/91] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:22.250973 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 49/92] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:22.253765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3ddfc6-aaf2-4f1f-9895-f42779e2b418 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:22.325070 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:22.388052 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3ddfc6-aaf2-4f1f-9895-f42779e2b418 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:22.430048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b022d66e-149d-458a-ad53-cbbfb7598e59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:22.430694 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b022d66e-149d-458a-ad53-cbbfb7598e59 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:22.430979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b022d66e-149d-458a-ad53-cbbfb7598e59 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:22.431256 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b022d66e-149d-458a-ad53-cbbfb7598e59 None admin] PATH_INFO: `/v3/users/7c12195059ec40bfafbb7be8e9f46734` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:22.432433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b022d66e-149d-458a-ad53-cbbfb7598e59 None admin] RBAC: Authorizing `identity:get_user(user_id=7c12195059ec40bfafbb7be8e9f46734)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:22.433883 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b022d66e-149d-458a-ad53-cbbfb7598e59 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:22.435487 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 44/93] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:22 2022] GET /identity/v3/users/7c12195059ec40bfafbb7be8e9f46734 => generated 247 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:22.441248 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e2fc9d-b97d-41de-a137-638691acd220 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:22.506981 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:22.568293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e2fc9d-b97d-41de-a137-638691acd220 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:22.591796 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b022d66e-149d-458a-ad53-cbbfb7598e59 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:22.634738 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d502831-f7a5-4899-8aa3-939be54c1a95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:22.635371 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d502831-f7a5-4899-8aa3-939be54c1a95 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:22.635686 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d502831-f7a5-4899-8aa3-939be54c1a95 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:22.635970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d502831-f7a5-4899-8aa3-939be54c1a95 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:22.756838 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b022d66e-149d-458a-ad53-cbbfb7598e59 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:22.782383 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:22.795044 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d502831-f7a5-4899-8aa3-939be54c1a95 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:22.796561 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d502831-f7a5-4899-8aa3-939be54c1a95 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:22.819329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a252e554-4ff1-4f01-9ad9-bf82e365fa9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:22.820009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a252e554-4ff1-4f01-9ad9-bf82e365fa9e None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:22.820295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a252e554-4ff1-4f01-9ad9-bf82e365fa9e None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:22.820574 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a252e554-4ff1-4f01-9ad9-bf82e365fa9e None admin] PATH_INFO: `/v3/roles/member` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:22.845925 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a252e554-4ff1-4f01-9ad9-bf82e365fa9e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:22.851647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-a252e554-4ff1-4f01-9ad9-bf82e365fa9e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:22.851647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:22.851647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:22.851647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:22.851647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:22.851647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:22.851647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:22.851647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:22.851647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:22.851647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:22.851647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:22.851647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:22.851647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:22.851647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:22.851647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:22.851647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:22.852537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:22.852537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:22.852537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:22.852537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:22.852537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:22.852537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:22.852537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:22.852537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:22.852537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:22.852537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:22.852537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:22.852537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:22.852537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:22.852537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:22.852537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:22.852537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:22.853297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:22.853297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:22.853297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:22.853297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:22.853297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:22.853297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:22.853297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:22.853297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:22.853297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:22.853297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:22.853297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:22.853297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:22.853297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 45/94] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:43:22.861861 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a252e554-4ff1-4f01-9ad9-bf82e365fa9e None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:22.903920 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:22.990806 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 50/95] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:43:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 550 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 31 14:43:22.992560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9ba825-5e1f-498c-978f-362a49ddeb69 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:22.993142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9ba825-5e1f-498c-978f-362a49ddeb69 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:22.993633 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9ba825-5e1f-498c-978f-362a49ddeb69 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:22.995307 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 51/96] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:22.996398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9ba825-5e1f-498c-978f-362a49ddeb69 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:23.048010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a252e554-4ff1-4f01-9ad9-bf82e365fa9e None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:23.112527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9ba825-5e1f-498c-978f-362a49ddeb69 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:23.136278 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14043a7f-4c61-4cf7-be57-2ccefc8fe21f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:23.136909 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14043a7f-4c61-4cf7-be57-2ccefc8fe21f None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:23.137220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14043a7f-4c61-4cf7-be57-2ccefc8fe21f None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:23.137499 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14043a7f-4c61-4cf7-be57-2ccefc8fe21f None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:23.138142 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14043a7f-4c61-4cf7-be57-2ccefc8fe21f None admin] RBAC: Adding query filter params (name=member) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:23.138446 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14043a7f-4c61-4cf7-be57-2ccefc8fe21f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:23.139863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14043a7f-4c61-4cf7-be57-2ccefc8fe21f None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:23.151546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a717e7-7ec8-40a5-9abb-bb940e39ac6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:23.152525 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a717e7-7ec8-40a5-9abb-bb940e39ac6a None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:23.153009 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a717e7-7ec8-40a5-9abb-bb940e39ac6a None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:23.153484 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a717e7-7ec8-40a5-9abb-bb940e39ac6a None admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:23.154815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a717e7-7ec8-40a5-9abb-bb940e39ac6a None admin] RBAC: Authorizing `identity:get_project(project_id=dc35ec7707ee4a09b3784f8b3e76686c)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:23.156491 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a717e7-7ec8-40a5-9abb-bb940e39ac6a None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:23.158548 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 52/97] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:22 2022] GET /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c => generated 298 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:23.160022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc80669-d3f5-4b9f-9a00-ad4f06df18e3 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:23.160515 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc80669-d3f5-4b9f-9a00-ad4f06df18e3 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:23.160991 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc80669-d3f5-4b9f-9a00-ad4f06df18e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:23.161097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 46/98] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:23.164374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14043a7f-4c61-4cf7-be57-2ccefc8fe21f None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:23.182969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-5fc80669-d3f5-4b9f-9a00-ad4f06df18e3 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:23.184922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5fc80669-d3f5-4b9f-9a00-ad4f06df18e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:23.217550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:23.220726 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:23.275313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 53/99] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:23.276567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc80669-d3f5-4b9f-9a00-ad4f06df18e3 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:23.301923 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14043a7f-4c61-4cf7-be57-2ccefc8fe21f None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:23.373677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1535ff5-d654-4329-965f-d0d6ad1cffc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:23.374414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1535ff5-d654-4329-965f-d0d6ad1cffc1 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:23.374699 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1535ff5-d654-4329-965f-d0d6ad1cffc1 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:23.374987 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1535ff5-d654-4329-965f-d0d6ad1cffc1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:23.376274 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1535ff5-d654-4329-965f-d0d6ad1cffc1 None admin] RBAC: Adding query filter params (role.id=57572a37675d4c418e13794a7b8f3a5d, scope.project.id=dc35ec7707ee4a09b3784f8b3e76686c, user.id=7c12195059ec40bfafbb7be8e9f46734) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:23.376579 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1535ff5-d654-4329-965f-d0d6ad1cffc1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:23.378075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1535ff5-d654-4329-965f-d0d6ad1cffc1 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:23.396444 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc80669-d3f5-4b9f-9a00-ad4f06df18e3 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:23.397934 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 47/100] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:43:23 2022] GET /identity/v3/role_assignments?user.id=7c12195059ec40bfafbb7be8e9f46734&scope.project.id=dc35ec7707ee4a09b3784f8b3e76686c&role.id=57572a37675d4c418e13794a7b8f3a5d => generated 608 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:23.399251 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f83fe67-74c2-49a3-812b-8350b75a73b1 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:23.399485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f83fe67-74c2-49a3-812b-8350b75a73b1 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:23.399835 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f83fe67-74c2-49a3-812b-8350b75a73b1 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:23.401577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 48/101] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:23.407098 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1eed97a-16f1-4313-9b4c-1e06f3625971 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:23.407468 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1eed97a-16f1-4313-9b4c-1e06f3625971 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:23.407786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1eed97a-16f1-4313-9b4c-1e06f3625971 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:23.440537 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4943702c-c857-4cda-87c8-4c52dcaa6ff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:23.441561 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4943702c-c857-4cda-87c8-4c52dcaa6ff2 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:23.442515 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4943702c-c857-4cda-87c8-4c52dcaa6ff2 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:23.442993 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4943702c-c857-4cda-87c8-4c52dcaa6ff2 None admin] PATH_INFO: `/v3/users/b0dd86f9993647219d6b755d3d7c09f5` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:23.444479 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4943702c-c857-4cda-87c8-4c52dcaa6ff2 None admin] RBAC: Authorizing `identity:get_user(user_id=b0dd86f9993647219d6b755d3d7c09f5)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:23.444593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-c1eed97a-16f1-4313-9b4c-1e06f3625971 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:23.445789 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-c1eed97a-16f1-4313-9b4c-1e06f3625971 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:23.446537 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4943702c-c857-4cda-87c8-4c52dcaa6ff2 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:23.448646 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 54/102] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:23 2022] GET /identity/v3/users/b0dd86f9993647219d6b755d3d7c09f5 => generated 275 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:23.457919 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4943702c-c857-4cda-87c8-4c52dcaa6ff2 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:23.508184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:23.580178 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 49/103] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:43:23.610421 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1eed97a-16f1-4313-9b4c-1e06f3625971 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:23.680071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:23.691750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4943702c-c857-4cda-87c8-4c52dcaa6ff2 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:23.745674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee59839-68e3-44d6-af6a-fce99bf60813 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:23.746292 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee59839-68e3-44d6-af6a-fce99bf60813 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:23.746573 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee59839-68e3-44d6-af6a-fce99bf60813 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:23.746844 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee59839-68e3-44d6-af6a-fce99bf60813 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:23.792486 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1eed97a-16f1-4313-9b4c-1e06f3625971 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:23.829683 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c66a927-a08a-4347-b36a-d5911836e391 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:23.830404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c66a927-a08a-4347-b36a-d5911836e391 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:23.830627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c66a927-a08a-4347-b36a-d5911836e391 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:23.830900 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c66a927-a08a-4347-b36a-d5911836e391 None admin] PATH_INFO: `/v3/roles/member` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:23.831138 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee59839-68e3-44d6-af6a-fce99bf60813 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:23.832862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee59839-68e3-44d6-af6a-fce99bf60813 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:23.842929 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c66a927-a08a-4347-b36a-d5911836e391 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:23.848287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-6c66a927-a08a-4347-b36a-d5911836e391 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:23.848287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:23.848287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:23.848287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:23.848287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:23.848287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:23.848287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:23.848287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:23.848287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:23.848287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:23.848287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:23.848287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:23.848287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:23.848287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:23.848287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:23.848287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:23.849325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:23.849325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:23.849325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:23.849325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:23.849325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:23.849325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:23.849325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:23.849325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:23.849325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:23.849325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:23.849325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:23.849325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:23.849325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:23.849325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:23.849325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:23.849325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:23.850225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:23.850225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:23.850225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:23.850225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:23.850225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:23.850225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:23.850225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:23.850225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:23.850225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:23.850225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:23.850225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:23.850225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:23.850225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 50/104] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:23.851006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2932a3a-9375-4e1a-9428-ce8088458466 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:23.851006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2932a3a-9375-4e1a-9428-ce8088458466 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:23.851143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2932a3a-9375-4e1a-9428-ce8088458466 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:23.856318 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 51/105] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:23 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:23.856318 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299a9e5f-4045-476d-b0b3-3607b07a549c None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:23.856318 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299a9e5f-4045-476d-b0b3-3607b07a549c None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:23.856318 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299a9e5f-4045-476d-b0b3-3607b07a549c None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:23.856978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 52/106] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:23.857706 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299a9e5f-4045-476d-b0b3-3607b07a549c None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:23.877653 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:23.954154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 55/107] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:43:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 497 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 31 14:43:23.955747 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414293b4-9779-4595-8d16-1ab7bdd15f83 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:23.956035 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414293b4-9779-4595-8d16-1ab7bdd15f83 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:23.956318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414293b4-9779-4595-8d16-1ab7bdd15f83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:23.962303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:23.970147 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299a9e5f-4045-476d-b0b3-3607b07a549c None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:23.974280 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-414293b4-9779-4595-8d16-1ab7bdd15f83 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:23.975491 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-414293b4-9779-4595-8d16-1ab7bdd15f83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:24.004149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d88ba5-dd86-4a9c-9fc6-a1c70f2c9a11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:24.004759 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d88ba5-dd86-4a9c-9fc6-a1c70f2c9a11 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:24.005046 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d88ba5-dd86-4a9c-9fc6-a1c70f2c9a11 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:24.005323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d88ba5-dd86-4a9c-9fc6-a1c70f2c9a11 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:24.005990 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d88ba5-dd86-4a9c-9fc6-a1c70f2c9a11 None admin] RBAC: Adding query filter params (name=member) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:24.006295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d88ba5-dd86-4a9c-9fc6-a1c70f2c9a11 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:24.007714 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d88ba5-dd86-4a9c-9fc6-a1c70f2c9a11 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:24.015575 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 53/108] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:24.016982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba32df3-a90e-455d-abd8-1eeaf9b1d8c8 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:24.017273 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba32df3-a90e-455d-abd8-1eeaf9b1d8c8 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:24.017551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba32df3-a90e-455d-abd8-1eeaf9b1d8c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:24.034639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-6ba32df3-a90e-455d-abd8-1eeaf9b1d8c8 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:24.035876 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6ba32df3-a90e-455d-abd8-1eeaf9b1d8c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:24.043186 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 56/109] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:24.044069 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414293b4-9779-4595-8d16-1ab7bdd15f83 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:24.105351 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 54/110] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:24.106601 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba32df3-a90e-455d-abd8-1eeaf9b1d8c8 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:24.120368 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:24.176159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414293b4-9779-4595-8d16-1ab7bdd15f83 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:24.223145 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a242d02f-c42c-4fe7-9385-8ec2a7ca7c7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:24.223939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a242d02f-c42c-4fe7-9385-8ec2a7ca7c7b None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:24.224295 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a242d02f-c42c-4fe7-9385-8ec2a7ca7c7b None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:24.224644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a242d02f-c42c-4fe7-9385-8ec2a7ca7c7b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:24.227236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:24.239298 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a242d02f-c42c-4fe7-9385-8ec2a7ca7c7b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:24.242011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba32df3-a90e-455d-abd8-1eeaf9b1d8c8 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:24.245516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-a242d02f-c42c-4fe7-9385-8ec2a7ca7c7b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:43:24.245516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:24.245516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:24.245516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:24.245516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:24.245516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:24.245516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:24.245516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:24.245516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:24.245516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:24.245516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:24.245516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:24.245516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:24.245516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:24.245516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:24.245516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:24.246369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:24.246369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:24.246369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:24.246369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:24.246369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:24.246369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:24.246369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:24.246369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:24.246369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:24.246369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:24.246369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:24.246369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:24.246369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:24.246369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:24.246369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:24.246369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:24.247114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:24.247114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:24.247114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:24.247114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:24.247114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:24.247114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:24.247114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:24.247114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:24.247114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:24.247114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:24.247114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:43:24.247114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:24.252009 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 57/111] 10.222.0.36 () {56 vars in 1114 bytes} [Mon Oct 31 14:43:24 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:24.252874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a242d02f-c42c-4fe7-9385-8ec2a7ca7c7b None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:24.282995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe978c7-066d-4b11-83cd-f76525c91126 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:24.284009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe978c7-066d-4b11-83cd-f76525c91126 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:24.284486 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe978c7-066d-4b11-83cd-f76525c91126 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:24.284949 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe978c7-066d-4b11-83cd-f76525c91126 None admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:24.286258 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe978c7-066d-4b11-83cd-f76525c91126 None admin] RBAC: Authorizing `identity:get_project(project_id=dc35ec7707ee4a09b3784f8b3e76686c)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:24.287978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe978c7-066d-4b11-83cd-f76525c91126 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:24.290163 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 55/112] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:24 2022] GET /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c => generated 298 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:24.291115 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe978c7-066d-4b11-83cd-f76525c91126 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:24.376326 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a242d02f-c42c-4fe7-9385-8ec2a7ca7c7b None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:24.390028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:24.398574 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe978c7-066d-4b11-83cd-f76525c91126 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:24.413279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5171198f-2ced-44cc-b9c4-8613351f8d5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:24.413909 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5171198f-2ced-44cc-b9c4-8613351f8d5d None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:24.414192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5171198f-2ced-44cc-b9c4-8613351f8d5d None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:24.414466 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5171198f-2ced-44cc-b9c4-8613351f8d5d None admin] PATH_INFO: `/v3/users/b0dd86f9993647219d6b755d3d7c09f5` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:24.415626 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5171198f-2ced-44cc-b9c4-8613351f8d5d None admin] RBAC: Authorizing `identity:get_user(user_id=b0dd86f9993647219d6b755d3d7c09f5)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:24.417068 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5171198f-2ced-44cc-b9c4-8613351f8d5d None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:24.418783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 58/113] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:24 2022] GET /identity/v3/users/b0dd86f9993647219d6b755d3d7c09f5 => generated 275 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:24.419409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5171198f-2ced-44cc-b9c4-8613351f8d5d None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:24.433366 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae53288-b625-4d5a-9cfb-4cf368c2e5e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:24.433972 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae53288-b625-4d5a-9cfb-4cf368c2e5e1 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:24.434263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae53288-b625-4d5a-9cfb-4cf368c2e5e1 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:24.434543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae53288-b625-4d5a-9cfb-4cf368c2e5e1 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:24.435214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae53288-b625-4d5a-9cfb-4cf368c2e5e1 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:24.435523 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae53288-b625-4d5a-9cfb-4cf368c2e5e1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:24.436941 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae53288-b625-4d5a-9cfb-4cf368c2e5e1 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:24.445466 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 56/114] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:43:24 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:24.446258 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae53288-b625-4d5a-9cfb-4cf368c2e5e1 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:24.465423 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:24.520657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5171198f-2ced-44cc-b9c4-8613351f8d5d None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:24.552423 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae53288-b625-4d5a-9cfb-4cf368c2e5e1 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:24.554639 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055232b2-db53-4dab-8552-79fa43e4cf79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:24.555245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055232b2-db53-4dab-8552-79fa43e4cf79 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:24.555525 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055232b2-db53-4dab-8552-79fa43e4cf79 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:24.555827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055232b2-db53-4dab-8552-79fa43e4cf79 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:24.556544 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055232b2-db53-4dab-8552-79fa43e4cf79 None admin] RBAC: Adding query filter params (role.id=bb354a7b796d467d91ff4b184011796e, scope.project.id=dc35ec7707ee4a09b3784f8b3e76686c, user.id=b0dd86f9993647219d6b755d3d7c09f5) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:24.556850 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055232b2-db53-4dab-8552-79fa43e4cf79 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:24.558272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055232b2-db53-4dab-8552-79fa43e4cf79 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:24.567805 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 59/115] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:43:24 2022] GET /identity/v3/role_assignments?user.id=b0dd86f9993647219d6b755d3d7c09f5&scope.project.id=dc35ec7707ee4a09b3784f8b3e76686c&role.id=bb354a7b796d467d91ff4b184011796e => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:24.568629 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055232b2-db53-4dab-8552-79fa43e4cf79 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:24.587250 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061b07ba-1d9f-4bdc-b2b4-d42400abe4b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:24.588193 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061b07ba-1d9f-4bdc-b2b4-d42400abe4b1 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:24.588309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061b07ba-1d9f-4bdc-b2b4-d42400abe4b1 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:24.588534 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061b07ba-1d9f-4bdc-b2b4-d42400abe4b1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:24.589742 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061b07ba-1d9f-4bdc-b2b4-d42400abe4b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:24.591260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061b07ba-1d9f-4bdc-b2b4-d42400abe4b1 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:24.592977 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 57/116] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:43:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:24.593801 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061b07ba-1d9f-4bdc-b2b4-d42400abe4b1 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:24.627463 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:24.690455 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:24.690996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055232b2-db53-4dab-8552-79fa43e4cf79 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:24.708880 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061b07ba-1d9f-4bdc-b2b4-d42400abe4b1 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:24.732015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7dd3348-cb68-4e5f-b8f9-4060881815bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:24.732358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dd3348-cb68-4e5f-b8f9-4060881815bc None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:24.732646 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dd3348-cb68-4e5f-b8f9-4060881815bc None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:24.732914 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dd3348-cb68-4e5f-b8f9-4060881815bc None admin] PATH_INFO: `/v3/roles/reader` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:24.746028 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c48cb0-a334-4fde-a8cd-383be7014a09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:24.746574 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7dd3348-cb68-4e5f-b8f9-4060881815bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:24.746684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c48cb0-a334-4fde-a8cd-383be7014a09 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:24.746907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c48cb0-a334-4fde-a8cd-383be7014a09 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:24.747181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c48cb0-a334-4fde-a8cd-383be7014a09 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:24.753586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-c7dd3348-cb68-4e5f-b8f9-4060881815bc None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 31 14:43:24.753586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:24.753586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:24.753586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:24.753586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:24.753586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:24.753586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:24.753586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:24.753586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:24.753586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:24.753586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:24.753586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:24.753586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:24.753586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:24.753586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:24.753586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:24.754441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:24.754441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:24.754441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:24.754441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:24.754441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:24.754441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:24.754441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:24.754441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:24.754441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:24.754441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:24.754441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:24.754441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:24.754441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:24.754441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:24.754441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:24.754441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:24.755185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:24.755185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:24.755185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:24.755185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:24.755185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:24.755185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:24.755185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:24.755185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:24.755185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:24.755185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:24.755185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 31 14:43:24.755185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:24.755185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 60/117] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:24 2022] GET /identity/v3/roles/reader => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:24.755866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7dd3348-cb68-4e5f-b8f9-4060881815bc None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:24.827413 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c48cb0-a334-4fde-a8cd-383be7014a09 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:24.829629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c48cb0-a334-4fde-a8cd-383be7014a09 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:24.862119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7dd3348-cb68-4e5f-b8f9-4060881815bc None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:24.864488 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:24.897100 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8047a9-5b0a-43f9-8c04-806eb36acf78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:24.897660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8047a9-5b0a-43f9-8c04-806eb36acf78 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:24.897944 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8047a9-5b0a-43f9-8c04-806eb36acf78 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:24.898218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8047a9-5b0a-43f9-8c04-806eb36acf78 None admin] PATH_INFO: `/v3/users/b0dd86f9993647219d6b755d3d7c09f5` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:24.899341 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8047a9-5b0a-43f9-8c04-806eb36acf78 None admin] RBAC: Authorizing `identity:get_user(user_id=b0dd86f9993647219d6b755d3d7c09f5)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:24.900827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8047a9-5b0a-43f9-8c04-806eb36acf78 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:24.902419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 61/118] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:24 2022] GET /identity/v3/users/b0dd86f9993647219d6b755d3d7c09f5 => generated 275 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:24.903848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8047a9-5b0a-43f9-8c04-806eb36acf78 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:24.912479 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:24.928221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 58/119] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:43:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 31 14:43:24.929044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c48cb0-a334-4fde-a8cd-383be7014a09 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:25.002211 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8047a9-5b0a-43f9-8c04-806eb36acf78 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:25.027146 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c48cb0-a334-4fde-a8cd-383be7014a09 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:25.035225 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2426dfae-698f-4adf-87d3-bd9bbc82830e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:25.035861 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2426dfae-698f-4adf-87d3-bd9bbc82830e None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:25.036151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2426dfae-698f-4adf-87d3-bd9bbc82830e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:25.036427 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2426dfae-698f-4adf-87d3-bd9bbc82830e None admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:25.037077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2426dfae-698f-4adf-87d3-bd9bbc82830e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:25.038522 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2426dfae-698f-4adf-87d3-bd9bbc82830e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:25.060572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473152d7-f5c6-49bf-9d26-aa59afa71a2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:25.061176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473152d7-f5c6-49bf-9d26-aa59afa71a2a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:25.061459 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473152d7-f5c6-49bf-9d26-aa59afa71a2a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:25.061735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473152d7-f5c6-49bf-9d26-aa59afa71a2a None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:25.062399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473152d7-f5c6-49bf-9d26-aa59afa71a2a None admin] RBAC: Adding query filter params (name=reader) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:25.062707 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473152d7-f5c6-49bf-9d26-aa59afa71a2a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:25.064637 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473152d7-f5c6-49bf-9d26-aa59afa71a2a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:25.067294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 62/120] 10.222.0.36 () {60 vars in 1141 bytes} [Mon Oct 31 14:43:24 2022] POST /identity/v3/projects => generated 302 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:43:25.068155 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2426dfae-698f-4adf-87d3-bd9bbc82830e None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:25.073279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 59/121] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:24 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:25.074197 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473152d7-f5c6-49bf-9d26-aa59afa71a2a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:25.101747 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:25.133123 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:25.193148 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473152d7-f5c6-49bf-9d26-aa59afa71a2a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:25.194093 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2426dfae-698f-4adf-87d3-bd9bbc82830e None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:25.232619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96ae64a-c9db-44a0-91c8-5469a62be1f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:25.233223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96ae64a-c9db-44a0-91c8-5469a62be1f2 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:25.233515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96ae64a-c9db-44a0-91c8-5469a62be1f2 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:25.233792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96ae64a-c9db-44a0-91c8-5469a62be1f2 None admin] PATH_INFO: `/v3/projects/5327d62517ed4878979f28cc0cd2e1bb` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:25.234948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96ae64a-c9db-44a0-91c8-5469a62be1f2 None admin] RBAC: Authorizing `identity:get_project(project_id=5327d62517ed4878979f28cc0cd2e1bb)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:25.236453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96ae64a-c9db-44a0-91c8-5469a62be1f2 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:25.236910 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d50918-72ef-47c5-887d-443e3b243531 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:25.237539 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d50918-72ef-47c5-887d-443e3b243531 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:25.237824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d50918-72ef-47c5-887d-443e3b243531 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:25.238104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d50918-72ef-47c5-887d-443e3b243531 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:25.238191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 60/122] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:25 2022] GET /identity/v3/projects/5327d62517ed4878979f28cc0cd2e1bb => generated 312 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:25.243762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96ae64a-c9db-44a0-91c8-5469a62be1f2 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:25.315629 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d50918-72ef-47c5-887d-443e3b243531 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:25.317218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d50918-72ef-47c5-887d-443e3b243531 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:25.343946 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:25.360791 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96ae64a-c9db-44a0-91c8-5469a62be1f2 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:25.379894 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:25.400968 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255fabb3-f256-417f-b1bc-5caf5b178a18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:25.401616 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255fabb3-f256-417f-b1bc-5caf5b178a18 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:25.401864 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255fabb3-f256-417f-b1bc-5caf5b178a18 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:25.402179 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255fabb3-f256-417f-b1bc-5caf5b178a18 None admin] PATH_INFO: `/v3/users/972cc83157b3439691c3b917cc3e4d4a` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:25.417753 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255fabb3-f256-417f-b1bc-5caf5b178a18 None admin] RBAC: Authorizing `identity:get_user(user_id=972cc83157b3439691c3b917cc3e4d4a)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:25.419222 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255fabb3-f256-417f-b1bc-5caf5b178a18 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:25.420985 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 61/123] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:25 2022] GET /identity/v3/users/972cc83157b3439691c3b917cc3e4d4a => generated 289 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:25.421754 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255fabb3-f256-417f-b1bc-5caf5b178a18 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:25.435759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 63/124] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:43:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 31 14:43:25.435759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d50918-72ef-47c5-887d-443e3b243531 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:25.536522 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255fabb3-f256-417f-b1bc-5caf5b178a18 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:25.553463 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d50918-72ef-47c5-887d-443e3b243531 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:25.572222 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e111cf-41a0-4b37-a979-b8b9edc24eca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:25.572831 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e111cf-41a0-4b37-a979-b8b9edc24eca None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:25.573111 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e111cf-41a0-4b37-a979-b8b9edc24eca None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:25.573384 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e111cf-41a0-4b37-a979-b8b9edc24eca None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:25.574122 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e111cf-41a0-4b37-a979-b8b9edc24eca None admin] RBAC: Adding query filter params (role.id=bb354a7b796d467d91ff4b184011796e, scope.project.id=5327d62517ed4878979f28cc0cd2e1bb, user.id=b0dd86f9993647219d6b755d3d7c09f5) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:25.574431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e111cf-41a0-4b37-a979-b8b9edc24eca None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:25.575963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e111cf-41a0-4b37-a979-b8b9edc24eca None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:25.580228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:25.591585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 62/125] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:43:25 2022] GET /identity/v3/role_assignments?user.id=b0dd86f9993647219d6b755d3d7c09f5&scope.project.id=5327d62517ed4878979f28cc0cd2e1bb&role.id=bb354a7b796d467d91ff4b184011796e => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:25.591585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e111cf-41a0-4b37-a979-b8b9edc24eca None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:25.603705 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aebfb1e-cd54-4980-a8be-fe58c0a396f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:25.604555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aebfb1e-cd54-4980-a8be-fe58c0a396f3 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:25.604947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aebfb1e-cd54-4980-a8be-fe58c0a396f3 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:25.605296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aebfb1e-cd54-4980-a8be-fe58c0a396f3 None admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:25.606674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aebfb1e-cd54-4980-a8be-fe58c0a396f3 None admin] RBAC: Authorizing `identity:get_project(project_id=dc35ec7707ee4a09b3784f8b3e76686c)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:25.608273 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aebfb1e-cd54-4980-a8be-fe58c0a396f3 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:25.610674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 64/126] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:25 2022] GET /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c => generated 298 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:25.616529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aebfb1e-cd54-4980-a8be-fe58c0a396f3 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:25.642643 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:25.726904 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e111cf-41a0-4b37-a979-b8b9edc24eca None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:25.742012 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aebfb1e-cd54-4980-a8be-fe58c0a396f3 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:25.773005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e09fe77-b79c-41a4-bf86-5681b3bdfc43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:25.774237 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e09fe77-b79c-41a4-bf86-5681b3bdfc43 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:25.774726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e09fe77-b79c-41a4-bf86-5681b3bdfc43 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:25.775305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e09fe77-b79c-41a4-bf86-5681b3bdfc43 None admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:25.777427 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e09fe77-b79c-41a4-bf86-5681b3bdfc43 None admin] RBAC: Authorizing `identity:get_project(project_id=dc35ec7707ee4a09b3784f8b3e76686c)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:25.779770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e09fe77-b79c-41a4-bf86-5681b3bdfc43 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:25.782748 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 63/127] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:25 2022] GET /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c => generated 298 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:25.788718 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37df4570-adfa-457e-80f3-51b5acd69ac1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:25.789462 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37df4570-adfa-457e-80f3-51b5acd69ac1 None admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:25.789752 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37df4570-adfa-457e-80f3-51b5acd69ac1 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:25.789902 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e09fe77-b79c-41a4-bf86-5681b3bdfc43 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:25.790030 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37df4570-adfa-457e-80f3-51b5acd69ac1 None admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c/users/972cc83157b3439691c3b917cc3e4d4a/roles/d5174defb0fa4869819ae433429cb2df` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:25.792064 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37df4570-adfa-457e-80f3-51b5acd69ac1 None admin] RBAC: Authorizing `identity:create_grant(project_id=dc35ec7707ee4a09b3784f8b3e76686c, user_id=972cc83157b3439691c3b917cc3e4d4a, role_id=d5174defb0fa4869819ae433429cb2df)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:25.793525 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37df4570-adfa-457e-80f3-51b5acd69ac1 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:25.814586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 65/128] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:43:25 2022] PUT /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c/users/972cc83157b3439691c3b917cc3e4d4a/roles/d5174defb0fa4869819ae433429cb2df => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:43:25.918686 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e09fe77-b79c-41a4-bf86-5681b3bdfc43 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:25.927940 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:25.958544 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9e2e56-d9a6-4335-8de2-17a9843db82d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:25.959165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9e2e56-d9a6-4335-8de2-17a9843db82d None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:25.959508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9e2e56-d9a6-4335-8de2-17a9843db82d None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:25.959816 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9e2e56-d9a6-4335-8de2-17a9843db82d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:25.960564 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9e2e56-d9a6-4335-8de2-17a9843db82d None admin] RBAC: Adding query filter params (role.id=addf152a48404c27b7d71386e81b8f76, scope.project.id=dc35ec7707ee4a09b3784f8b3e76686c, user.id=b0dd86f9993647219d6b755d3d7c09f5) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:25.960881 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9e2e56-d9a6-4335-8de2-17a9843db82d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:25.962355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9e2e56-d9a6-4335-8de2-17a9843db82d None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:25.969352 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 64/129] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:43:25 2022] GET /identity/v3/role_assignments?user.id=b0dd86f9993647219d6b755d3d7c09f5&scope.project.id=dc35ec7707ee4a09b3784f8b3e76686c&role.id=addf152a48404c27b7d71386e81b8f76 => generated 608 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:27.298274 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c922f4e-bb3b-4312-bb63-37bd66faf28c None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:27.298905 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c922f4e-bb3b-4312-bb63-37bd66faf28c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:27.298905 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c922f4e-bb3b-4312-bb63-37bd66faf28c None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:27.300191 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 66/130] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:27.305399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a95697-987d-453e-842b-b42ed230ca15 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:27.305809 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a95697-987d-453e-842b-b42ed230ca15 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:27.306035 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a95697-987d-453e-842b-b42ed230ca15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:27.324074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-72a95697-987d-453e-842b-b42ed230ca15 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:27.325242 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-72a95697-987d-453e-842b-b42ed230ca15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:27.390529 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 65/131] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:27.662639 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c922f4e-bb3b-4312-bb63-37bd66faf28c None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:27.716941 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:27.768188 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c922f4e-bb3b-4312-bb63-37bd66faf28c None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:27.801874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7dc274-a1e5-44d4-8b89-e32c84c744fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:27.802504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7dc274-a1e5-44d4-8b89-e32c84c744fc None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:27.802789 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7dc274-a1e5-44d4-8b89-e32c84c744fc None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:27.803064 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7dc274-a1e5-44d4-8b89-e32c84c744fc None admin] PATH_INFO: `/v3/domains/default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:27.804190 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7dc274-a1e5-44d4-8b89-e32c84c744fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:27.805641 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7dc274-a1e5-44d4-8b89-e32c84c744fc None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:27.807233 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 67/132] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:43:27 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:27.812755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a95697-987d-453e-842b-b42ed230ca15 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:27.911925 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:27.914960 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a95697-987d-453e-842b-b42ed230ca15 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:27.950682 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7dce7a-64b4-4242-844d-6e7f88815968 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:27.951286 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7dce7a-64b4-4242-844d-6e7f88815968 None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:27.951584 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7dce7a-64b4-4242-844d-6e7f88815968 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:27.951869 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7dce7a-64b4-4242-844d-6e7f88815968 None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:27.952520 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7dce7a-64b4-4242-844d-6e7f88815968 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:27.954185 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7dce7a-64b4-4242-844d-6e7f88815968 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:27.974959 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 66/133] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:43:27 2022] POST /identity/v3/users => generated 283 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:43:27.994308 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda1e9d7-8b29-405e-9622-09ad67d928f4 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:27.994685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda1e9d7-8b29-405e-9622-09ad67d928f4 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:27.995026 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda1e9d7-8b29-405e-9622-09ad67d928f4 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:27.996544 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 68/134] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:28.002518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc8900e-4e06-41b1-9b00-b5e6118e4731 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:28.002810 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc8900e-4e06-41b1-9b00-b5e6118e4731 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:28.003091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc8900e-4e06-41b1-9b00-b5e6118e4731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:28.022624 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-5fc8900e-4e06-41b1-9b00-b5e6118e4731 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:28.023867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5fc8900e-4e06-41b1-9b00-b5e6118e4731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:28.094610 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 67/135] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:28.370675 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda1e9d7-8b29-405e-9622-09ad67d928f4 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:28.481372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda1e9d7-8b29-405e-9622-09ad67d928f4 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:28.502339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:28.522068 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afda98b-d211-4bcf-a511-078509be09a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:28.522068 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afda98b-d211-4bcf-a511-078509be09a1 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:28.522068 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afda98b-d211-4bcf-a511-078509be09a1 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:28.522392 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afda98b-d211-4bcf-a511-078509be09a1 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:28.535578 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afda98b-d211-4bcf-a511-078509be09a1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:28.539842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-2afda98b-d211-4bcf-a511-078509be09a1 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 31 14:43:28.539842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:28.539842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:28.539842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:28.539842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:28.539842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:28.539842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:28.539842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:28.539842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:28.539842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:28.539842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:28.539842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:28.539842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:28.539842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:28.539842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:28.539842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:28.540731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:28.540731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:28.540731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:28.540731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:28.540731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:28.540731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:28.540731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:28.540731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:28.540731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:28.540731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:28.540731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:28.540731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:28.540731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:28.540731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:28.540731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:28.540731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:28.541516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:28.541516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:28.541516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:28.541516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:28.541516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:28.541516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:28.541516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:28.541516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:28.541516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:28.541516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:28.541516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 31 14:43:28.541516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:28.541516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 69/136] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:28 2022] GET /identity/v3/roles/reader => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:28.548268 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc8900e-4e06-41b1-9b00-b5e6118e4731 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:28.648061 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:28.659509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc8900e-4e06-41b1-9b00-b5e6118e4731 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:28.698534 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa80c5d-d99a-48c2-b089-7a66b6a58b6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:28.699146 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa80c5d-d99a-48c2-b089-7a66b6a58b6f None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:28.699432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa80c5d-d99a-48c2-b089-7a66b6a58b6f None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:28.699745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa80c5d-d99a-48c2-b089-7a66b6a58b6f None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:28.700483 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa80c5d-d99a-48c2-b089-7a66b6a58b6f None admin] RBAC: Adding query filter params (name=reader) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:28.700795 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa80c5d-d99a-48c2-b089-7a66b6a58b6f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:28.702181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa80c5d-d99a-48c2-b089-7a66b6a58b6f None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:28.711949 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 68/137] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:28 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:28.717340 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afda98b-d211-4bcf-a511-078509be09a1 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:28.829866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afda98b-d211-4bcf-a511-078509be09a1 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:28.865663 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b615eda5-c3b6-40c3-87ad-cf551549b015 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:28.866314 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b615eda5-c3b6-40c3-87ad-cf551549b015 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:28.866598 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b615eda5-c3b6-40c3-87ad-cf551549b015 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:28.866876 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b615eda5-c3b6-40c3-87ad-cf551549b015 None admin] PATH_INFO: `/v3/users/972cc83157b3439691c3b917cc3e4d4a` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:28.868054 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b615eda5-c3b6-40c3-87ad-cf551549b015 None admin] RBAC: Authorizing `identity:get_user(user_id=972cc83157b3439691c3b917cc3e4d4a)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:28.869511 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b615eda5-c3b6-40c3-87ad-cf551549b015 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:28.871103 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 70/138] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:28 2022] GET /identity/v3/users/972cc83157b3439691c3b917cc3e4d4a => generated 289 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:28.877289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa80c5d-d99a-48c2-b089-7a66b6a58b6f None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:28.986917 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa80c5d-d99a-48c2-b089-7a66b6a58b6f None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:29.024417 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34545df2-4a46-47e3-8e86-44bf6c363b1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:29.025021 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34545df2-4a46-47e3-8e86-44bf6c363b1e None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:29.025300 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34545df2-4a46-47e3-8e86-44bf6c363b1e None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:29.025575 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34545df2-4a46-47e3-8e86-44bf6c363b1e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:29.044508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:29.100377 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34545df2-4a46-47e3-8e86-44bf6c363b1e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:29.101844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34545df2-4a46-47e3-8e86-44bf6c363b1e None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:29.210311 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 69/139] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:43:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 31 14:43:29.215916 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b615eda5-c3b6-40c3-87ad-cf551549b015 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:29.268097 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:29.321130 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b615eda5-c3b6-40c3-87ad-cf551549b015 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:29.357177 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9be6a8-9b61-4f6d-95ad-2c60a7477b9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:29.357818 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9be6a8-9b61-4f6d-95ad-2c60a7477b9a None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:29.358102 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9be6a8-9b61-4f6d-95ad-2c60a7477b9a None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:29.358381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9be6a8-9b61-4f6d-95ad-2c60a7477b9a None admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:29.360199 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9be6a8-9b61-4f6d-95ad-2c60a7477b9a None admin] RBAC: Authorizing `identity:get_project(project_id=dc35ec7707ee4a09b3784f8b3e76686c)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:29.361643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9be6a8-9b61-4f6d-95ad-2c60a7477b9a None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:29.363600 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 71/140] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:29 2022] GET /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c => generated 298 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:29.372626 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34545df2-4a46-47e3-8e86-44bf6c363b1e None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:29.424711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:29.478684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34545df2-4a46-47e3-8e86-44bf6c363b1e None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:29.514263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0275d5-98a5-4772-9b08-505a2910470e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:29.514866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0275d5-98a5-4772-9b08-505a2910470e None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:29.515145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0275d5-98a5-4772-9b08-505a2910470e None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:29.515417 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0275d5-98a5-4772-9b08-505a2910470e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:29.516295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0275d5-98a5-4772-9b08-505a2910470e None admin] RBAC: Adding query filter params (role.id=d5174defb0fa4869819ae433429cb2df, scope.project.id=dc35ec7707ee4a09b3784f8b3e76686c, user.id=972cc83157b3439691c3b917cc3e4d4a) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:29.516608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0275d5-98a5-4772-9b08-505a2910470e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:29.520770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0275d5-98a5-4772-9b08-505a2910470e None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:29.524469 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 70/141] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:43:29 2022] GET /identity/v3/role_assignments?user.id=972cc83157b3439691c3b917cc3e4d4a&scope.project.id=dc35ec7707ee4a09b3784f8b3e76686c&role.id=d5174defb0fa4869819ae433429cb2df => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:30.299348 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2d2447-9f98-44d5-a0bd-eb5fa3c95e05 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:30.299930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2d2447-9f98-44d5-a0bd-eb5fa3c95e05 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:30.299930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2d2447-9f98-44d5-a0bd-eb5fa3c95e05 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:30.301423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 72/142] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:30.306458 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff443ef-d878-41f7-a140-6f820237144b None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:30.306840 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff443ef-d878-41f7-a140-6f820237144b None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:30.307018 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff443ef-d878-41f7-a140-6f820237144b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:30.327072 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-6ff443ef-d878-41f7-a140-6f820237144b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:30.329529 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6ff443ef-d878-41f7-a140-6f820237144b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:30.404641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 71/143] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:30.458406 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0d0188-c73c-426f-90c8-e8f7eb3a15a0 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:30.458758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0d0188-c73c-426f-90c8-e8f7eb3a15a0 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:30.459255 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0d0188-c73c-426f-90c8-e8f7eb3a15a0 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:30.460733 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 73/144] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:30.466012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd3f6a2-adf5-4d49-8557-8073d3cba2cf None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:30.466307 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd3f6a2-adf5-4d49-8557-8073d3cba2cf None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:30.466586 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd3f6a2-adf5-4d49-8557-8073d3cba2cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:30.485950 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-5fd3f6a2-adf5-4d49-8557-8073d3cba2cf None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:30.487150 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5fd3f6a2-adf5-4d49-8557-8073d3cba2cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:30.519485 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f926062-6cf6-4c47-8f82-cce522c51eec None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:30.520151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f926062-6cf6-4c47-8f82-cce522c51eec None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:30.520707 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f926062-6cf6-4c47-8f82-cce522c51eec None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:30.522231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 74/145] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:30.527527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1155618b-b350-4c8f-ab88-875d254241bb None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:30.527907 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1155618b-b350-4c8f-ab88-875d254241bb None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:30.530618 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1155618b-b350-4c8f-ab88-875d254241bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:30.534469 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:30.548323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-1155618b-b350-4c8f-ab88-875d254241bb None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:30.549924 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1155618b-b350-4c8f-ab88-875d254241bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:30.567972 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 72/146] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:30.626343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 75/147] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:43:30.674175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd3f6a2-adf5-4d49-8557-8073d3cba2cf None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:30.775637 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd3f6a2-adf5-4d49-8557-8073d3cba2cf None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:30.810559 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5dc4af7-9abf-492f-899b-558512386cbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:30.811121 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5dc4af7-9abf-492f-899b-558512386cbf None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:30.811409 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5dc4af7-9abf-492f-899b-558512386cbf None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:30.811714 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5dc4af7-9abf-492f-899b-558512386cbf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:30.822209 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5dc4af7-9abf-492f-899b-558512386cbf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:30.827113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-a5dc4af7-9abf-492f-899b-558512386cbf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:43:30.827113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:30.827113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:30.827113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:30.827113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:30.827113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:30.827113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:30.827113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:30.827113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:30.827113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:30.827113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:30.827113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:30.827113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:30.827113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:30.827113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:30.827113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:30.828674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:30.828674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:30.828674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:30.828674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:30.828674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:30.828674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:30.828674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:30.828674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:30.828674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:30.828674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:30.828674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:30.828674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:30.828674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:30.828674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:30.828674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:30.828674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:30.829540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:30.829540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:30.829540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:30.829540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:30.829540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:30.829540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:30.829540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:30.829540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:30.829540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:30.829540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:30.829540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:43:30.829540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:30.829540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 73/148] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:43:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:30.836075 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1155618b-b350-4c8f-ab88-875d254241bb None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:30.841500 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5dc4af7-9abf-492f-899b-558512386cbf None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:30.875834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:30.888182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:30.941400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1155618b-b350-4c8f-ab88-875d254241bb None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:30.946529 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5dc4af7-9abf-492f-899b-558512386cbf None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:30.975355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262d8ee0-c69d-4ae1-9e9c-975614de8e5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:30.976307 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262d8ee0-c69d-4ae1-9e9c-975614de8e5d None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:30.976590 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262d8ee0-c69d-4ae1-9e9c-975614de8e5d None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:30.976870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262d8ee0-c69d-4ae1-9e9c-975614de8e5d None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:30.977526 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-262d8ee0-c69d-4ae1-9e9c-975614de8e5d None admin] RBAC: Adding query filter params (name=admin) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:30.977830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-262d8ee0-c69d-4ae1-9e9c-975614de8e5d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:30.979197 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-262d8ee0-c69d-4ae1-9e9c-975614de8e5d None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:30.979966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f37f39-c5f5-492e-a5f4-2135a51f06de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:30.980583 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f37f39-c5f5-492e-a5f4-2135a51f06de None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:30.980857 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f37f39-c5f5-492e-a5f4-2135a51f06de None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:30.981362 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f37f39-c5f5-492e-a5f4-2135a51f06de None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:30.987650 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 76/149] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:30.991591 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f37f39-c5f5-492e-a5f4-2135a51f06de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:30.992575 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262d8ee0-c69d-4ae1-9e9c-975614de8e5d None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:30.996420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-13f37f39-c5f5-492e-a5f4-2135a51f06de None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:43:30.996420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:30.996420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:30.996420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:30.996420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:30.996420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:30.996420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:30.996420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:30.996420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:30.996420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:30.996420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:30.996420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:30.996420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:30.996420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:30.996420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:30.996420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:30.997460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:30.997460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:30.997460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:30.997460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:30.997460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:30.997460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:30.997460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:30.997460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:30.997460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:30.997460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:30.997460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:30.997460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:30.997460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:30.997460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:30.997460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:30.997460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:30.998410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:30.998410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:30.998410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:30.998410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:30.998410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:30.998410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:30.998410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:30.998410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:30.998410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:30.998410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:30.998410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:43:30.998410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:30.998410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 74/150] 10.222.0.36 () {56 vars in 1114 bytes} [Mon Oct 31 14:43:30 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:43:30.998410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f37f39-c5f5-492e-a5f4-2135a51f06de None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:31.086157 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:31.094309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262d8ee0-c69d-4ae1-9e9c-975614de8e5d None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:31.095802 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f37f39-c5f5-492e-a5f4-2135a51f06de None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:31.103175 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:31.130101 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01018e5-103f-458d-9f11-63d32180bcef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:31.130622 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d028e8e-1e0d-406d-bbb0-eaa507cffd8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:31.130731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01018e5-103f-458d-9f11-63d32180bcef None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:31.131000 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01018e5-103f-458d-9f11-63d32180bcef None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:31.131235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d028e8e-1e0d-406d-bbb0-eaa507cffd8f None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:31.131325 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01018e5-103f-458d-9f11-63d32180bcef None admin] PATH_INFO: `/v3/domains/default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:31.131510 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d028e8e-1e0d-406d-bbb0-eaa507cffd8f None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:31.131823 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d028e8e-1e0d-406d-bbb0-eaa507cffd8f None admin] PATH_INFO: `/v3/users/85816fdbf93142779ff5a42a18b11db7` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:31.132410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01018e5-103f-458d-9f11-63d32180bcef None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:31.133870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01018e5-103f-458d-9f11-63d32180bcef None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:31.135445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 77/151] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:43:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:31.136249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01018e5-103f-458d-9f11-63d32180bcef None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:31.145987 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d028e8e-1e0d-406d-bbb0-eaa507cffd8f None admin] RBAC: Authorizing `identity:get_user(user_id=85816fdbf93142779ff5a42a18b11db7)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:31.147771 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d028e8e-1e0d-406d-bbb0-eaa507cffd8f None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:31.149409 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 75/152] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:30 2022] GET /identity/v3/users/85816fdbf93142779ff5a42a18b11db7 => generated 283 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:31.149409 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d028e8e-1e0d-406d-bbb0-eaa507cffd8f None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:31.239793 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01018e5-103f-458d-9f11-63d32180bcef None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:31.248894 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d028e8e-1e0d-406d-bbb0-eaa507cffd8f None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:31.274033 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91786a9-9ba0-4dbe-83bb-bc52776ba18d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:31.274644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91786a9-9ba0-4dbe-83bb-bc52776ba18d None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:31.274926 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91786a9-9ba0-4dbe-83bb-bc52776ba18d None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:31.275244 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91786a9-9ba0-4dbe-83bb-bc52776ba18d None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:31.275939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91786a9-9ba0-4dbe-83bb-bc52776ba18d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:31.276242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91786a9-9ba0-4dbe-83bb-bc52776ba18d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:31.277611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91786a9-9ba0-4dbe-83bb-bc52776ba18d None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:31.281782 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e83de5-e1c7-44d2-bf5f-d88c9141077e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:31.282368 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e83de5-e1c7-44d2-bf5f-d88c9141077e None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:31.282645 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e83de5-e1c7-44d2-bf5f-d88c9141077e None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:31.282913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e83de5-e1c7-44d2-bf5f-d88c9141077e None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:31.283573 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e83de5-e1c7-44d2-bf5f-d88c9141077e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:31.285006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e83de5-e1c7-44d2-bf5f-d88c9141077e None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:31.285365 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 78/153] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:43:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:31.286244 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91786a9-9ba0-4dbe-83bb-bc52776ba18d None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:31.314530 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 76/154] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:43:31 2022] POST /identity/v3/users => generated 297 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:43:31.324549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e83de5-e1c7-44d2-bf5f-d88c9141077e None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:31.328115 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:31.344078 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:31.400009 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91786a9-9ba0-4dbe-83bb-bc52776ba18d None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:31.440618 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dd2111-939e-49e5-89c5-8c613336834d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:31.441272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dd2111-939e-49e5-89c5-8c613336834d None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:31.441556 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dd2111-939e-49e5-89c5-8c613336834d None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:31.441833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dd2111-939e-49e5-89c5-8c613336834d None admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:31.443682 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dd2111-939e-49e5-89c5-8c613336834d None admin] RBAC: Authorizing `identity:get_project(project_id=5847498d7fa84446a3389fbb99498368)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:31.445335 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dd2111-939e-49e5-89c5-8c613336834d None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:31.447273 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 79/155] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:31 2022] GET /identity/v3/projects/5847498d7fa84446a3389fbb99498368 => generated 302 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:31.453250 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dd2111-939e-49e5-89c5-8c613336834d None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:31.456892 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e83de5-e1c7-44d2-bf5f-d88c9141077e None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:31.495087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d910391a-9807-4e7e-8225-d2b9087801e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:31.495759 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d910391a-9807-4e7e-8225-d2b9087801e7 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:31.496040 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d910391a-9807-4e7e-8225-d2b9087801e7 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:31.496853 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d910391a-9807-4e7e-8225-d2b9087801e7 None admin] PATH_INFO: `/v3/users/85816fdbf93142779ff5a42a18b11db7` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:31.497459 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d910391a-9807-4e7e-8225-d2b9087801e7 None admin] RBAC: Authorizing `identity:get_user(user_id=85816fdbf93142779ff5a42a18b11db7)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:31.498907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d910391a-9807-4e7e-8225-d2b9087801e7 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:31.500747 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 77/156] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:31 2022] GET /identity/v3/users/85816fdbf93142779ff5a42a18b11db7 => generated 283 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:31.506209 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d910391a-9807-4e7e-8225-d2b9087801e7 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:31.575258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dd2111-939e-49e5-89c5-8c613336834d None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:31.586492 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:31.604518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:31.623867 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39046053-6594-41db-b13c-bddadda6b63e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:31.624539 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39046053-6594-41db-b13c-bddadda6b63e None admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:31.625334 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39046053-6594-41db-b13c-bddadda6b63e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:31.625679 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39046053-6594-41db-b13c-bddadda6b63e None admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368/users/85816fdbf93142779ff5a42a18b11db7/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:31.628809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39046053-6594-41db-b13c-bddadda6b63e None admin] RBAC: Authorizing `identity:create_grant(project_id=5847498d7fa84446a3389fbb99498368, user_id=85816fdbf93142779ff5a42a18b11db7, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:31.630108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39046053-6594-41db-b13c-bddadda6b63e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:31.636395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d910391a-9807-4e7e-8225-d2b9087801e7 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:31.648675 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 80/157] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:43:31 2022] PUT /identity/v3/projects/5847498d7fa84446a3389fbb99498368/users/85816fdbf93142779ff5a42a18b11db7/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:43:31.685769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7befd1e9-b025-4fce-9001-ec21e7f088cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:31.686428 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7befd1e9-b025-4fce-9001-ec21e7f088cd None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:31.686705 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7befd1e9-b025-4fce-9001-ec21e7f088cd None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:31.686980 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7befd1e9-b025-4fce-9001-ec21e7f088cd None admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:31.688791 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7befd1e9-b025-4fce-9001-ec21e7f088cd None admin] RBAC: Authorizing `identity:get_project(project_id=5847498d7fa84446a3389fbb99498368)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:31.690275 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7befd1e9-b025-4fce-9001-ec21e7f088cd None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:31.692398 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 78/158] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:31 2022] GET /identity/v3/projects/5847498d7fa84446a3389fbb99498368 => generated 302 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:31.698068 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39046053-6594-41db-b13c-bddadda6b63e None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:31.815501 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39046053-6594-41db-b13c-bddadda6b63e None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:31.852874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7d0272-c150-41f8-9fdf-0db4f9dd7dc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:31.853541 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7d0272-c150-41f8-9fdf-0db4f9dd7dc0 None admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:31.853823 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7d0272-c150-41f8-9fdf-0db4f9dd7dc0 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:31.854100 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7d0272-c150-41f8-9fdf-0db4f9dd7dc0 None admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368/users/85816fdbf93142779ff5a42a18b11db7/roles/addf152a48404c27b7d71386e81b8f76` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:31.856330 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7d0272-c150-41f8-9fdf-0db4f9dd7dc0 None admin] RBAC: Authorizing `identity:create_grant(project_id=5847498d7fa84446a3389fbb99498368, user_id=85816fdbf93142779ff5a42a18b11db7, role_id=addf152a48404c27b7d71386e81b8f76)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:31.857721 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7d0272-c150-41f8-9fdf-0db4f9dd7dc0 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:31.873816 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 81/159] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:43:31 2022] PUT /identity/v3/projects/5847498d7fa84446a3389fbb99498368/users/85816fdbf93142779ff5a42a18b11db7/roles/addf152a48404c27b7d71386e81b8f76 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:43:33.607516 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a8b6c0-417e-4d17-9c2a-777c64d77be2 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:33.608403 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a8b6c0-417e-4d17-9c2a-777c64d77be2 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:33.608819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a8b6c0-417e-4d17-9c2a-777c64d77be2 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:33.611390 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 79/160] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:33 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:33.618998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37a6787-381e-4f18-8c02-a0147c9a8a0c None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:33.619514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37a6787-381e-4f18-8c02-a0147c9a8a0c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:33.619643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37a6787-381e-4f18-8c02-a0147c9a8a0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:33.640810 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-c37a6787-381e-4f18-8c02-a0147c9a8a0c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:33.642307 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-c37a6787-381e-4f18-8c02-a0147c9a8a0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:33.664097 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:33.713401 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 82/161] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:33.730396 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb3908f-cff0-4406-a25a-0221f3e6c2e8 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:33.730676 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb3908f-cff0-4406-a25a-0221f3e6c2e8 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:33.730955 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb3908f-cff0-4406-a25a-0221f3e6c2e8 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:33.732366 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 80/162] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:33.738472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0992278-b661-410d-bcf6-e609d1c43c0c None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:33.738765 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0992278-b661-410d-bcf6-e609d1c43c0c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:33.739041 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0992278-b661-410d-bcf6-e609d1c43c0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:33.758304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-f0992278-b661-410d-bcf6-e609d1c43c0c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:33.759489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-f0992278-b661-410d-bcf6-e609d1c43c0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:33.833264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 83/163] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:33.910961 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e08f3e3-9a6d-4127-9c09-81244c8ac111 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:33.911257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e08f3e3-9a6d-4127-9c09-81244c8ac111 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:33.911508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e08f3e3-9a6d-4127-9c09-81244c8ac111 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:33.912902 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 81/164] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:33.918159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74a9d53-3537-462a-b0bc-cbfd9efe60fe None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:33.918625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74a9d53-3537-462a-b0bc-cbfd9efe60fe None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:33.919037 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74a9d53-3537-462a-b0bc-cbfd9efe60fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:33.938460 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-a74a9d53-3537-462a-b0bc-cbfd9efe60fe None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:33.939891 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a74a9d53-3537-462a-b0bc-cbfd9efe60fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:33.991285 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e08f3e3-9a6d-4127-9c09-81244c8ac111 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:34.014566 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 84/165] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:34.100507 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b051f7-0827-461d-9172-3767cb949aac None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:34.100807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b051f7-0827-461d-9172-3767cb949aac None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:34.100987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b051f7-0827-461d-9172-3767cb949aac None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:34.102331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 85/166] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:43:34.107501 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e08f3e3-9a6d-4127-9c09-81244c8ac111 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:34.107958 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee94d9b7-c5d9-4ad3-894c-b62362abaec8 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:34.108435 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee94d9b7-c5d9-4ad3-894c-b62362abaec8 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:34.108902 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee94d9b7-c5d9-4ad3-894c-b62362abaec8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:34.128947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-ee94d9b7-c5d9-4ad3-894c-b62362abaec8 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:34.130253 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ee94d9b7-c5d9-4ad3-894c-b62362abaec8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:34.142965 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:34.144677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7645ff-0dc0-4c78-a87c-3b9795fdc592 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:34.144808 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7645ff-0dc0-4c78-a87c-3b9795fdc592 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:34.145007 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7645ff-0dc0-4c78-a87c-3b9795fdc592 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:34.145269 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7645ff-0dc0-4c78-a87c-3b9795fdc592 None admin] PATH_INFO: `/v3/roles/member` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:34.156353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7645ff-0dc0-4c78-a87c-3b9795fdc592 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:34.161845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-8b7645ff-0dc0-4c78-a87c-3b9795fdc592 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:34.161845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:34.161845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:34.161845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:34.161845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:34.161845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:34.161845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:34.161845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:34.161845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:34.161845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:34.161845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:34.161845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:34.161845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:34.161845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:34.161845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:34.161845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:34.162721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:34.162721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:34.162721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:34.162721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:34.162721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:34.162721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:34.162721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:34.162721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:34.162721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:34.162721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:34.162721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:34.162721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:34.162721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:34.162721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:34.162721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:34.162721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:34.163515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:34.163515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:34.163515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:34.163515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:34.163515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:34.163515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:34.163515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:34.163515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:34.163515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:34.163515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:34.163515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:34.163515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:34.163515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 82/167] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:43:34.164620 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7645ff-0dc0-4c78-a87c-3b9795fdc592 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:34.170508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:34.212078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 86/168] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:43:34.217481 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee94d9b7-c5d9-4ad3-894c-b62362abaec8 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:34.282153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7645ff-0dc0-4c78-a87c-3b9795fdc592 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:34.319633 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e9d817-bace-4ab7-bbc8-c4ce73959969 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:34.320263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e9d817-bace-4ab7-bbc8-c4ce73959969 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:34.320540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e9d817-bace-4ab7-bbc8-c4ce73959969 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:34.320812 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e9d817-bace-4ab7-bbc8-c4ce73959969 None admin] PATH_INFO: `/v3/domains/default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:34.322261 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e9d817-bace-4ab7-bbc8-c4ce73959969 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:34.324098 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e9d817-bace-4ab7-bbc8-c4ce73959969 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:34.325856 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 83/169] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:43:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:34.326524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e9d817-bace-4ab7-bbc8-c4ce73959969 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:34.351576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee94d9b7-c5d9-4ad3-894c-b62362abaec8 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:34.389090 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f24114-ad9a-4140-a152-ad1d27965d9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:34.389696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f24114-ad9a-4140-a152-ad1d27965d9b None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:34.389981 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f24114-ad9a-4140-a152-ad1d27965d9b None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:34.390314 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f24114-ad9a-4140-a152-ad1d27965d9b None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:34.392369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f24114-ad9a-4140-a152-ad1d27965d9b None admin] RBAC: Adding query filter params (name=member) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:34.392674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f24114-ad9a-4140-a152-ad1d27965d9b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:34.394048 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f24114-ad9a-4140-a152-ad1d27965d9b None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:34.398594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:34.403796 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 87/170] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:34 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:34.409250 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f24114-ad9a-4140-a152-ad1d27965d9b None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:34.421166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:34.432589 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e9d817-bace-4ab7-bbc8-c4ce73959969 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:34.467412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0f0b1c-4477-47e7-b1b3-1336728d9a70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:34.468044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f0b1c-4477-47e7-b1b3-1336728d9a70 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:34.468321 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f0b1c-4477-47e7-b1b3-1336728d9a70 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:34.468619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f0b1c-4477-47e7-b1b3-1336728d9a70 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:34.479670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0f0b1c-4477-47e7-b1b3-1336728d9a70 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:34.486441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-ee0f0b1c-4477-47e7-b1b3-1336728d9a70 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:43:34.486441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:34.486441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:34.486441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:34.486441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:34.486441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:34.486441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:34.486441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:34.486441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:34.486441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:34.486441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:34.486441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:34.486441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:34.486441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:34.486441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:34.486441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:34.487325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:34.487325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:34.487325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:34.487325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:34.487325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:34.487325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:34.487325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:34.487325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:34.487325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:34.487325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:34.487325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:34.487325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:34.487325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:34.487325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:34.487325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:34.487325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:34.488165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:34.488165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:34.488165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:34.488165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:34.488165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:34.488165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:34.488165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:34.488165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:34.488165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:34.488165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:34.488165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:43:34.488165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:34.488165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 84/171] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:43:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:34.488819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0f0b1c-4477-47e7-b1b3-1336728d9a70 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:34.519088 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f24114-ad9a-4140-a152-ad1d27965d9b None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:34.553409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f13b46-b804-4079-b423-66f749af0fd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:34.554015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f13b46-b804-4079-b423-66f749af0fd1 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:34.554289 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f13b46-b804-4079-b423-66f749af0fd1 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:34.554560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f13b46-b804-4079-b423-66f749af0fd1 None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:34.555199 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f13b46-b804-4079-b423-66f749af0fd1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:34.556651 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f13b46-b804-4079-b423-66f749af0fd1 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:34.576556 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 88/172] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:43:34 2022] POST /identity/v3/users => generated 297 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:43:34.577359 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f13b46-b804-4079-b423-66f749af0fd1 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:34.598300 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0f0b1c-4477-47e7-b1b3-1336728d9a70 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:34.630957 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:34.639301 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dcac58-ac5d-44c2-b86f-563b30dc5274 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:34.639986 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dcac58-ac5d-44c2-b86f-563b30dc5274 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:34.640276 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dcac58-ac5d-44c2-b86f-563b30dc5274 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:34.641096 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dcac58-ac5d-44c2-b86f-563b30dc5274 None admin] PATH_INFO: `/v3/users/15f8e6d81e974150908237a2bd902f3b` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:34.651979 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:34.657502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07dcac58-ac5d-44c2-b86f-563b30dc5274 None admin] RBAC: Authorizing `identity:get_user(user_id=15f8e6d81e974150908237a2bd902f3b)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:34.658996 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07dcac58-ac5d-44c2-b86f-563b30dc5274 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:34.660801 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 85/173] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:34 2022] GET /identity/v3/users/15f8e6d81e974150908237a2bd902f3b => generated 297 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:34.660801 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dcac58-ac5d-44c2-b86f-563b30dc5274 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:34.699918 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f13b46-b804-4079-b423-66f749af0fd1 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:34.738605 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9162ce7d-f5f2-48aa-bdc1-4ed5ef273789 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:34.739258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9162ce7d-f5f2-48aa-bdc1-4ed5ef273789 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:34.739539 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9162ce7d-f5f2-48aa-bdc1-4ed5ef273789 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:34.739839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9162ce7d-f5f2-48aa-bdc1-4ed5ef273789 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:34.751710 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9162ce7d-f5f2-48aa-bdc1-4ed5ef273789 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:34.757327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-9162ce7d-f5f2-48aa-bdc1-4ed5ef273789 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:43:34.757327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:34.757327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:34.757327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:34.757327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:34.757327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:34.757327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:34.757327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:34.757327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:34.757327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:34.757327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:34.757327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:34.757327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:34.757327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:34.757327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:34.757327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:34.758172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:34.758172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:34.758172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:34.758172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:34.758172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:34.758172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:34.758172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:34.758172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:34.758172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:34.758172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:34.758172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:34.758172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:34.758172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:34.758172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:34.758172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:34.758172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:34.758928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:34.758928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:34.758928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:34.758928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:34.758928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:34.758928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:34.758928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:34.758928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:34.758928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:34.758928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:34.758928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:43:34.758928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:34.759497 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 89/174] 10.222.0.36 () {56 vars in 1114 bytes} [Mon Oct 31 14:43:34 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:34.763927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9162ce7d-f5f2-48aa-bdc1-4ed5ef273789 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:34.789008 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dcac58-ac5d-44c2-b86f-563b30dc5274 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:34.829829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0c36de-4cda-48f0-a05b-391e1bd9623f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:34.830447 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0c36de-4cda-48f0-a05b-391e1bd9623f None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:34.830723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0c36de-4cda-48f0-a05b-391e1bd9623f None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:34.830989 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0c36de-4cda-48f0-a05b-391e1bd9623f None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:34.832280 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a0c36de-4cda-48f0-a05b-391e1bd9623f None admin] RBAC: Adding query filter params (name=admin) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:34.832280 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a0c36de-4cda-48f0-a05b-391e1bd9623f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:34.833461 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a0c36de-4cda-48f0-a05b-391e1bd9623f None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:34.843996 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 86/175] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:34.847593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0c36de-4cda-48f0-a05b-391e1bd9623f None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:34.899240 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9162ce7d-f5f2-48aa-bdc1-4ed5ef273789 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:34.907170 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:34.929259 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:34.941736 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83980f72-a6c0-4a5d-bfb6-75ba32e05881 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:34.942372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83980f72-a6c0-4a5d-bfb6-75ba32e05881 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:34.942645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83980f72-a6c0-4a5d-bfb6-75ba32e05881 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:34.942912 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83980f72-a6c0-4a5d-bfb6-75ba32e05881 None admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:34.944594 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83980f72-a6c0-4a5d-bfb6-75ba32e05881 None admin] RBAC: Authorizing `identity:get_project(project_id=5847498d7fa84446a3389fbb99498368)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:34.945497 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83980f72-a6c0-4a5d-bfb6-75ba32e05881 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:34.947304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 90/176] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:34 2022] GET /identity/v3/projects/5847498d7fa84446a3389fbb99498368 => generated 302 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:34.952492 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83980f72-a6c0-4a5d-bfb6-75ba32e05881 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:34.976962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0c36de-4cda-48f0-a05b-391e1bd9623f None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:35.013248 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cd4f3a-6502-4c4d-bf09-a6e1506c7abe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:35.014104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cd4f3a-6502-4c4d-bf09-a6e1506c7abe None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:35.014524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cd4f3a-6502-4c4d-bf09-a6e1506c7abe None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:35.014975 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cd4f3a-6502-4c4d-bf09-a6e1506c7abe None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:35.015892 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cd4f3a-6502-4c4d-bf09-a6e1506c7abe None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:35.016365 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cd4f3a-6502-4c4d-bf09-a6e1506c7abe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:35.017918 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cd4f3a-6502-4c4d-bf09-a6e1506c7abe None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:35.026704 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 87/177] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:43:34 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:35.031922 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cd4f3a-6502-4c4d-bf09-a6e1506c7abe None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:35.083098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83980f72-a6c0-4a5d-bfb6-75ba32e05881 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:35.124065 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57560ca-34ef-48b1-a222-c1bc935ba4ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:35.125389 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57560ca-34ef-48b1-a222-c1bc935ba4ba None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:35.125693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57560ca-34ef-48b1-a222-c1bc935ba4ba None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:35.126032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57560ca-34ef-48b1-a222-c1bc935ba4ba None admin] PATH_INFO: `/v3/users/85816fdbf93142779ff5a42a18b11db7` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:35.127274 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57560ca-34ef-48b1-a222-c1bc935ba4ba None admin] RBAC: Authorizing `identity:get_user(user_id=85816fdbf93142779ff5a42a18b11db7)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:35.129600 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57560ca-34ef-48b1-a222-c1bc935ba4ba None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:35.131686 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 91/178] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:34 2022] GET /identity/v3/users/85816fdbf93142779ff5a42a18b11db7 => generated 283 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:35.135722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57560ca-34ef-48b1-a222-c1bc935ba4ba None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:35.159212 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cd4f3a-6502-4c4d-bf09-a6e1506c7abe None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:35.176777 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:35.200005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689232df-bcbe-4207-978b-1fc914a1a1ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:35.200667 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689232df-bcbe-4207-978b-1fc914a1a1ee None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:35.200955 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689232df-bcbe-4207-978b-1fc914a1a1ee None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:35.201224 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689232df-bcbe-4207-978b-1fc914a1a1ee None admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368/users/15f8e6d81e974150908237a2bd902f3b/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:35.203422 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689232df-bcbe-4207-978b-1fc914a1a1ee None admin] RBAC: Authorizing `identity:create_grant(project_id=5847498d7fa84446a3389fbb99498368, user_id=15f8e6d81e974150908237a2bd902f3b, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:35.204921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689232df-bcbe-4207-978b-1fc914a1a1ee None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:35.214938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:35.220294 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 88/179] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:43:35 2022] PUT /identity/v3/projects/5847498d7fa84446a3389fbb99498368/users/15f8e6d81e974150908237a2bd902f3b/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:43:35.221575 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689232df-bcbe-4207-978b-1fc914a1a1ee None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:35.267478 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57560ca-34ef-48b1-a222-c1bc935ba4ba None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:35.315643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42dda3e4-7f34-4596-94fa-d620a13a9819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:35.316294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dda3e4-7f34-4596-94fa-d620a13a9819 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:35.317006 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dda3e4-7f34-4596-94fa-d620a13a9819 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:35.317006 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dda3e4-7f34-4596-94fa-d620a13a9819 None admin] PATH_INFO: `/v3/users/85816fdbf93142779ff5a42a18b11db7` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:35.318160 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42dda3e4-7f34-4596-94fa-d620a13a9819 None admin] RBAC: Authorizing `identity:get_user(user_id=85816fdbf93142779ff5a42a18b11db7)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:35.319660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42dda3e4-7f34-4596-94fa-d620a13a9819 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:35.321600 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 92/180] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:35 2022] GET /identity/v3/users/85816fdbf93142779ff5a42a18b11db7 => generated 283 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:35.327345 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42dda3e4-7f34-4596-94fa-d620a13a9819 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:35.401728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689232df-bcbe-4207-978b-1fc914a1a1ee None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:35.471812 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1cda96-b779-4898-8586-d06ddc393b40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:35.472524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1cda96-b779-4898-8586-d06ddc393b40 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:35.472997 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1cda96-b779-4898-8586-d06ddc393b40 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:35.473334 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1cda96-b779-4898-8586-d06ddc393b40 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:35.493066 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42dda3e4-7f34-4596-94fa-d620a13a9819 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:35.523692 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:35.529347 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:35.541968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d79569-eaa7-484a-983b-3f94d661e389 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:35.542690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d79569-eaa7-484a-983b-3f94d661e389 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:35.543050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d79569-eaa7-484a-983b-3f94d661e389 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:35.543392 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d79569-eaa7-484a-983b-3f94d661e389 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:35.590694 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1cda96-b779-4898-8586-d06ddc393b40 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:35.592219 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1cda96-b779-4898-8586-d06ddc393b40 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:35.668039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d79569-eaa7-484a-983b-3f94d661e389 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:35.669697 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d79569-eaa7-484a-983b-3f94d661e389 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:35.797944 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 89/181] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:43:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 577 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 31 14:43:35.804233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1cda96-b779-4898-8586-d06ddc393b40 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:35.860895 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 93/182] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:43:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 534 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 31 14:43:35.866518 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d79569-eaa7-484a-983b-3f94d661e389 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:35.906900 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:35.913900 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:35.943484 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1cda96-b779-4898-8586-d06ddc393b40 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:35.988986 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbec48a1-9dee-437e-a0f2-073159f3284b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:35.989756 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbec48a1-9dee-437e-a0f2-073159f3284b None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:35.990036 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbec48a1-9dee-437e-a0f2-073159f3284b None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:35.990307 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbec48a1-9dee-437e-a0f2-073159f3284b None admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:35.994717 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbec48a1-9dee-437e-a0f2-073159f3284b None admin] RBAC: Authorizing `identity:get_project(project_id=5847498d7fa84446a3389fbb99498368)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:35.996307 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbec48a1-9dee-437e-a0f2-073159f3284b None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:35.998367 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 90/183] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:35 2022] GET /identity/v3/projects/5847498d7fa84446a3389fbb99498368 => generated 302 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:36.004451 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbec48a1-9dee-437e-a0f2-073159f3284b None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:36.020289 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d79569-eaa7-484a-983b-3f94d661e389 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:36.065849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8598d9a5-b921-4ce0-ad75-609db47a28c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:36.066417 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8598d9a5-b921-4ce0-ad75-609db47a28c3 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:36.066695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8598d9a5-b921-4ce0-ad75-609db47a28c3 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:36.066963 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8598d9a5-b921-4ce0-ad75-609db47a28c3 None admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:36.069118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8598d9a5-b921-4ce0-ad75-609db47a28c3 None admin] RBAC: Authorizing `identity:get_project(project_id=5847498d7fa84446a3389fbb99498368)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:36.070585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8598d9a5-b921-4ce0-ad75-609db47a28c3 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:36.072642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 94/184] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:35 2022] GET /identity/v3/projects/5847498d7fa84446a3389fbb99498368 => generated 302 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:36.079014 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8598d9a5-b921-4ce0-ad75-609db47a28c3 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:36.131341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbec48a1-9dee-437e-a0f2-073159f3284b None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:36.172336 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aadc17f-d6af-4c92-919f-224cd7761302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:36.173039 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aadc17f-d6af-4c92-919f-224cd7761302 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:36.173323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aadc17f-d6af-4c92-919f-224cd7761302 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:36.173655 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aadc17f-d6af-4c92-919f-224cd7761302 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:36.174388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aadc17f-d6af-4c92-919f-224cd7761302 None admin] RBAC: Adding query filter params (role.id=57572a37675d4c418e13794a7b8f3a5d, scope.project.id=5847498d7fa84446a3389fbb99498368, user.id=85816fdbf93142779ff5a42a18b11db7) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:36.174698 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aadc17f-d6af-4c92-919f-224cd7761302 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:36.177215 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aadc17f-d6af-4c92-919f-224cd7761302 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:36.186150 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 91/185] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:43:36 2022] GET /identity/v3/role_assignments?user.id=85816fdbf93142779ff5a42a18b11db7&scope.project.id=5847498d7fa84446a3389fbb99498368&role.id=57572a37675d4c418e13794a7b8f3a5d => generated 608 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:36.254877 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:36.262609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8598d9a5-b921-4ce0-ad75-609db47a28c3 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:36.311020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4da14e2-00a6-4395-a05e-9a4db726aa41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:36.311658 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4da14e2-00a6-4395-a05e-9a4db726aa41 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:36.311943 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4da14e2-00a6-4395-a05e-9a4db726aa41 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:36.313274 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4da14e2-00a6-4395-a05e-9a4db726aa41 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:36.314020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4da14e2-00a6-4395-a05e-9a4db726aa41 None admin] RBAC: Adding query filter params (role.id=addf152a48404c27b7d71386e81b8f76, scope.project.id=5847498d7fa84446a3389fbb99498368, user.id=85816fdbf93142779ff5a42a18b11db7) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:36.314333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4da14e2-00a6-4395-a05e-9a4db726aa41 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:36.315813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4da14e2-00a6-4395-a05e-9a4db726aa41 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:36.324590 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 95/186] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:43:36 2022] GET /identity/v3/role_assignments?user.id=85816fdbf93142779ff5a42a18b11db7&scope.project.id=5847498d7fa84446a3389fbb99498368&role.id=addf152a48404c27b7d71386e81b8f76 => generated 608 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:37.083920 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66554f53-b879-4a81-ab2f-264123921a35 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:37.084449 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66554f53-b879-4a81-ab2f-264123921a35 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:37.084449 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66554f53-b879-4a81-ab2f-264123921a35 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:37.085580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7800573-20d2-4549-93f9-3fef75004298 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:37.085983 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7800573-20d2-4549-93f9-3fef75004298 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:37.086150 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 92/187] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:37.086254 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7800573-20d2-4549-93f9-3fef75004298 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:37.087509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 96/188] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:37.091130 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa04c2a0-5da1-4a05-9bb7-f5f0180fb648 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:37.091371 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa04c2a0-5da1-4a05-9bb7-f5f0180fb648 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:37.091675 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa04c2a0-5da1-4a05-9bb7-f5f0180fb648 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:37.092272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca46bd4-1eb1-43c3-9e1c-778db91237be None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:37.092555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca46bd4-1eb1-43c3-9e1c-778db91237be None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:37.092827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca46bd4-1eb1-43c3-9e1c-778db91237be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:37.100716 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:37.112884 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-7ca46bd4-1eb1-43c3-9e1c-778db91237be None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:37.113244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-aa04c2a0-5da1-4a05-9bb7-f5f0180fb648 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:37.114066 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7ca46bd4-1eb1-43c3-9e1c-778db91237be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:37.114403 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-aa04c2a0-5da1-4a05-9bb7-f5f0180fb648 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:37.186799 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 93/189] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:37.188528 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 97/190] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:43:37.457432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa04c2a0-5da1-4a05-9bb7-f5f0180fb648 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:37.461204 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca46bd4-1eb1-43c3-9e1c-778db91237be None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:37.562305 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:37.567261 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa04c2a0-5da1-4a05-9bb7-f5f0180fb648 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:37.573159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca46bd4-1eb1-43c3-9e1c-778db91237be None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:37.602423 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c0a01c-0ca1-430a-86e6-bd925cb790b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:37.603054 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c0a01c-0ca1-430a-86e6-bd925cb790b1 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:37.603333 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c0a01c-0ca1-430a-86e6-bd925cb790b1 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:37.603630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c0a01c-0ca1-430a-86e6-bd925cb790b1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:37.604738 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c0a01c-0ca1-430a-86e6-bd925cb790b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:37.606169 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c0a01c-0ca1-430a-86e6-bd925cb790b1 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:37.607774 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 94/191] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:43:37 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:37.611662 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9afeda1-c120-4f63-8c4f-ed40e9593348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:37.612561 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9afeda1-c120-4f63-8c4f-ed40e9593348 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:37.612989 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9afeda1-c120-4f63-8c4f-ed40e9593348 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:37.613268 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9afeda1-c120-4f63-8c4f-ed40e9593348 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:37.613584 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c0a01c-0ca1-430a-86e6-bd925cb790b1 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:37.616811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:37.624269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9afeda1-c120-4f63-8c4f-ed40e9593348 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:37.629815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-b9afeda1-c120-4f63-8c4f-ed40e9593348 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 31 14:43:37.629815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:37.629815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:37.629815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:37.629815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:37.629815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:37.629815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:37.629815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:37.629815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:37.629815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:37.629815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:37.629815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:37.629815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:37.629815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:37.629815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:37.629815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:37.630661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:37.630661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:37.630661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:37.630661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:37.630661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:37.630661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:37.630661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:37.630661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:37.630661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:37.630661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:37.630661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:37.630661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:37.630661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:37.630661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:37.630661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:37.630661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:37.631421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:37.631421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:37.631421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:37.631421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:37.631421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:37.631421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:37.631421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:37.631421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:37.631421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:37.631421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:37.631421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 31 14:43:37.631421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:37.631421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 98/192] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:37 2022] GET /identity/v3/roles/reader => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:43:37.638150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9afeda1-c120-4f63-8c4f-ed40e9593348 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:37.729488 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c0a01c-0ca1-430a-86e6-bd925cb790b1 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:37.748107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9afeda1-c120-4f63-8c4f-ed40e9593348 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:37.766011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5558ad9-d4e6-40aa-8cbe-92cef851a5da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:37.766620 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5558ad9-d4e6-40aa-8cbe-92cef851a5da None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:37.766896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5558ad9-d4e6-40aa-8cbe-92cef851a5da None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:37.767164 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5558ad9-d4e6-40aa-8cbe-92cef851a5da None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:37.767844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5558ad9-d4e6-40aa-8cbe-92cef851a5da None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:37.769280 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5558ad9-d4e6-40aa-8cbe-92cef851a5da None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:37.786273 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406687b6-eac9-46e4-a077-244fa886b77e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:37.786884 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406687b6-eac9-46e4-a077-244fa886b77e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:37.787157 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406687b6-eac9-46e4-a077-244fa886b77e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:37.787425 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406687b6-eac9-46e4-a077-244fa886b77e None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:37.788113 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406687b6-eac9-46e4-a077-244fa886b77e None admin] RBAC: Adding query filter params (name=reader) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:37.788415 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406687b6-eac9-46e4-a077-244fa886b77e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:37.789790 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406687b6-eac9-46e4-a077-244fa886b77e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:37.790894 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 95/193] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:43:37 2022] POST /identity/v3/users => generated 293 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:43:37.792151 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f2e432-566d-444c-8faf-5f54938ed79a None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:37.792439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f2e432-566d-444c-8faf-5f54938ed79a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:37.792712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f2e432-566d-444c-8faf-5f54938ed79a None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:37.793927 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 96/194] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:37 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:37.800042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f9f772-e641-4d34-a5b0-09f36be79e30 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:37.800320 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f9f772-e641-4d34-a5b0-09f36be79e30 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:37.800597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f9f772-e641-4d34-a5b0-09f36be79e30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:37.811136 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 99/195] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:37 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:37.817668 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406687b6-eac9-46e4-a077-244fa886b77e None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:37.820724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-10f9f772-e641-4d34-a5b0-09f36be79e30 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:37.822481 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-10f9f772-e641-4d34-a5b0-09f36be79e30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:37.836303 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:37.887763 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:37.903820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 97/196] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:43:37.934371 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406687b6-eac9-46e4-a077-244fa886b77e None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:37.974218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdd53fb-0086-4fbf-8978-2bd1ffaecfff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:37.974947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdd53fb-0086-4fbf-8978-2bd1ffaecfff None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:37.975224 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdd53fb-0086-4fbf-8978-2bd1ffaecfff None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:37.975495 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdd53fb-0086-4fbf-8978-2bd1ffaecfff None admin] PATH_INFO: `/v3/users/e780cb5a88c341ada14aa47d4b51353e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:37.989972 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdd53fb-0086-4fbf-8978-2bd1ffaecfff None admin] RBAC: Authorizing `identity:get_user(user_id=e780cb5a88c341ada14aa47d4b51353e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:37.991621 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdd53fb-0086-4fbf-8978-2bd1ffaecfff None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:37.993402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 100/197] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:37 2022] GET /identity/v3/users/e780cb5a88c341ada14aa47d4b51353e => generated 297 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:37.999054 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f9f772-e641-4d34-a5b0-09f36be79e30 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:38.108273 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f9f772-e641-4d34-a5b0-09f36be79e30 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:38.149079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4665a3-92f9-4bb9-b4aa-8ff771618d59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:38.149954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4665a3-92f9-4bb9-b4aa-8ff771618d59 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:38.150234 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4665a3-92f9-4bb9-b4aa-8ff771618d59 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:38.150509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4665a3-92f9-4bb9-b4aa-8ff771618d59 None admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:38.151690 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4665a3-92f9-4bb9-b4aa-8ff771618d59 None admin] RBAC: Authorizing `identity:get_project(project_id=5847498d7fa84446a3389fbb99498368)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:38.153196 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4665a3-92f9-4bb9-b4aa-8ff771618d59 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:38.154854 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 98/198] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:37 2022] GET /identity/v3/projects/5847498d7fa84446a3389fbb99498368 => generated 302 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:38.161372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdd53fb-0086-4fbf-8978-2bd1ffaecfff None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:38.179930 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4665a3-92f9-4bb9-b4aa-8ff771618d59 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:38.242894 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:38.258578 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:38.290943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4665a3-92f9-4bb9-b4aa-8ff771618d59 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:38.303351 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdd53fb-0086-4fbf-8978-2bd1ffaecfff None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:38.329947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be31703-df05-4962-a06f-b7e2c772606b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:38.330751 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be31703-df05-4962-a06f-b7e2c772606b None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:38.331127 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be31703-df05-4962-a06f-b7e2c772606b None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:38.331494 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be31703-df05-4962-a06f-b7e2c772606b None admin] PATH_INFO: `/v3/roles/member` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:38.342773 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be31703-df05-4962-a06f-b7e2c772606b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:38.347626 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6100427b-7de9-43bb-9f6d-eecfc462ea5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:38.348300 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6100427b-7de9-43bb-9f6d-eecfc462ea5a None admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:38.348586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6100427b-7de9-43bb-9f6d-eecfc462ea5a None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:38.348865 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6100427b-7de9-43bb-9f6d-eecfc462ea5a None admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368/users/e780cb5a88c341ada14aa47d4b51353e/roles/d5174defb0fa4869819ae433429cb2df` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:38.349642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-0be31703-df05-4962-a06f-b7e2c772606b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:38.349642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:38.349642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:38.349642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:38.349642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:38.349642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:38.349642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:38.349642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:38.349642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:38.349642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:38.349642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:38.349642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:38.349642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:38.349642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:38.349642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:38.349642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:38.350737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:38.350737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:38.350737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:38.350737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:38.350737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:38.350737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:38.350737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:38.350737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:38.350737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:38.350737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:38.350737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:38.350737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:38.350737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:38.350737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:38.350737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:38.350737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:38.351770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:38.351770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:38.351770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:38.351770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:38.351770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:38.351770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:38.351770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:38.351770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:38.351770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:38.351770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:38.351770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:38.351770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:38.353360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 99/199] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:43:38.353919 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6100427b-7de9-43bb-9f6d-eecfc462ea5a None admin] RBAC: Authorizing `identity:create_grant(project_id=5847498d7fa84446a3389fbb99498368, user_id=e780cb5a88c341ada14aa47d4b51353e, role_id=d5174defb0fa4869819ae433429cb2df)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:38.355586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6100427b-7de9-43bb-9f6d-eecfc462ea5a None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:38.360145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be31703-df05-4962-a06f-b7e2c772606b None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:38.372971 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 101/200] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:43:38 2022] PUT /identity/v3/projects/5847498d7fa84446a3389fbb99498368/users/e780cb5a88c341ada14aa47d4b51353e/roles/d5174defb0fa4869819ae433429cb2df => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:43:38.483332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be31703-df05-4962-a06f-b7e2c772606b None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:38.502191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:38.523567 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995fe627-e276-481f-a3f1-3387f54d7f0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:38.524428 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995fe627-e276-481f-a3f1-3387f54d7f0c None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:38.524874 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995fe627-e276-481f-a3f1-3387f54d7f0c None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:38.525308 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995fe627-e276-481f-a3f1-3387f54d7f0c None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:38.526159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995fe627-e276-481f-a3f1-3387f54d7f0c None admin] RBAC: Adding query filter params (name=member) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:38.526622 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995fe627-e276-481f-a3f1-3387f54d7f0c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:38.528223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995fe627-e276-481f-a3f1-3387f54d7f0c None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:38.537842 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 100/201] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:38.543780 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6100427b-7de9-43bb-9f6d-eecfc462ea5a None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:38.646690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6100427b-7de9-43bb-9f6d-eecfc462ea5a None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:38.676028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:38.682144 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75582b39-8d30-48a9-abdb-fe82ab3b2086 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:38.682786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75582b39-8d30-48a9-abdb-fe82ab3b2086 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:38.683095 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75582b39-8d30-48a9-abdb-fe82ab3b2086 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:38.683380 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75582b39-8d30-48a9-abdb-fe82ab3b2086 None admin] PATH_INFO: `/v3/users/15f8e6d81e974150908237a2bd902f3b` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:38.684588 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75582b39-8d30-48a9-abdb-fe82ab3b2086 None admin] RBAC: Authorizing `identity:get_user(user_id=15f8e6d81e974150908237a2bd902f3b)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:38.686254 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75582b39-8d30-48a9-abdb-fe82ab3b2086 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:38.688786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 102/202] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:38 2022] GET /identity/v3/users/15f8e6d81e974150908237a2bd902f3b => generated 297 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:38.694384 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995fe627-e276-481f-a3f1-3387f54d7f0c None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:38.809785 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995fe627-e276-481f-a3f1-3387f54d7f0c None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:38.845220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d40cda8-f663-4011-82d9-68d30736fc29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:38.846086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d40cda8-f663-4011-82d9-68d30736fc29 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:38.846376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d40cda8-f663-4011-82d9-68d30736fc29 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:38.846649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d40cda8-f663-4011-82d9-68d30736fc29 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:38.899586 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:38.928999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d40cda8-f663-4011-82d9-68d30736fc29 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:38.930823 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d40cda8-f663-4011-82d9-68d30736fc29 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:39.038230 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 101/203] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:43:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 31 14:43:39.044228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75582b39-8d30-48a9-abdb-fe82ab3b2086 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:39.150281 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75582b39-8d30-48a9-abdb-fe82ab3b2086 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:39.187346 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4946a9f9-ff72-4869-b350-102c041c52bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:39.188804 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4946a9f9-ff72-4869-b350-102c041c52bd None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:39.189282 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4946a9f9-ff72-4869-b350-102c041c52bd None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:39.189780 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4946a9f9-ff72-4869-b350-102c041c52bd None admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:39.191149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4946a9f9-ff72-4869-b350-102c041c52bd None admin] RBAC: Authorizing `identity:get_project(project_id=5847498d7fa84446a3389fbb99498368)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:39.192822 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4946a9f9-ff72-4869-b350-102c041c52bd None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:39.195014 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 103/204] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:39 2022] GET /identity/v3/projects/5847498d7fa84446a3389fbb99498368 => generated 302 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:39.207606 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d40cda8-f663-4011-82d9-68d30736fc29 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:39.287262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:39.311594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d40cda8-f663-4011-82d9-68d30736fc29 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:39.355015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e70b6ae-0308-4af1-9d97-f9f324246ad8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:39.355647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e70b6ae-0308-4af1-9d97-f9f324246ad8 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:39.355931 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e70b6ae-0308-4af1-9d97-f9f324246ad8 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:39.356206 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e70b6ae-0308-4af1-9d97-f9f324246ad8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:39.356933 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e70b6ae-0308-4af1-9d97-f9f324246ad8 None admin] RBAC: Adding query filter params (role.id=bb354a7b796d467d91ff4b184011796e, scope.project.id=5847498d7fa84446a3389fbb99498368, user.id=15f8e6d81e974150908237a2bd902f3b) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:39.357276 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e70b6ae-0308-4af1-9d97-f9f324246ad8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:39.358986 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e70b6ae-0308-4af1-9d97-f9f324246ad8 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:39.365418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 102/205] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:43:39 2022] GET /identity/v3/role_assignments?user.id=15f8e6d81e974150908237a2bd902f3b&scope.project.id=5847498d7fa84446a3389fbb99498368&role.id=bb354a7b796d467d91ff4b184011796e => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:40.223694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb646e3c-4126-4ad3-825e-03fa05d7a136 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:40.224212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb646e3c-4126-4ad3-825e-03fa05d7a136 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:40.224286 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb646e3c-4126-4ad3-825e-03fa05d7a136 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:40.225628 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 104/206] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:40.230822 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce840564-e8a6-4991-ac9f-00c912f41d40 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:40.231262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce840564-e8a6-4991-ac9f-00c912f41d40 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:40.231403 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce840564-e8a6-4991-ac9f-00c912f41d40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:40.250832 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-ce840564-e8a6-4991-ac9f-00c912f41d40 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:40.252037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ce840564-e8a6-4991-ac9f-00c912f41d40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:40.273927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98eabff-1523-4710-9530-dbc4dea60f2a None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:40.274254 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98eabff-1523-4710-9530-dbc4dea60f2a None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:40.274486 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98eabff-1523-4710-9530-dbc4dea60f2a None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:40.276052 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 105/207] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:43:40.281413 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22db80f-c2e3-4a4c-8d39-6e98e278b7b3 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:40.281694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22db80f-c2e3-4a4c-8d39-6e98e278b7b3 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:40.281964 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22db80f-c2e3-4a4c-8d39-6e98e278b7b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:40.300457 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-b22db80f-c2e3-4a4c-8d39-6e98e278b7b3 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:40.301641 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b22db80f-c2e3-4a4c-8d39-6e98e278b7b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:40.327584 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 103/208] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:40.362976 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:40.371335 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 106/209] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:43:40.594258 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce840564-e8a6-4991-ac9f-00c912f41d40 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:40.642081 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22db80f-c2e3-4a4c-8d39-6e98e278b7b3 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:40.667497 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:40.698011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce840564-e8a6-4991-ac9f-00c912f41d40 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:40.731165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20d4f9f-edc5-424a-8d88-a07faf8dd1db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:40.731809 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20d4f9f-edc5-424a-8d88-a07faf8dd1db None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:40.732086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20d4f9f-edc5-424a-8d88-a07faf8dd1db None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:40.732355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20d4f9f-edc5-424a-8d88-a07faf8dd1db None admin] PATH_INFO: `/v3/domains/default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:40.733441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b20d4f9f-edc5-424a-8d88-a07faf8dd1db None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:40.735664 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b20d4f9f-edc5-424a-8d88-a07faf8dd1db None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:40.737236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 104/210] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:43:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:40.738336 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92da738b-34a7-4c18-8e65-512974f66c58 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:40.738660 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92da738b-34a7-4c18-8e65-512974f66c58 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:40.738923 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92da738b-34a7-4c18-8e65-512974f66c58 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:40.740216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 105/211] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:40 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:40.742853 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92da738b-34a7-4c18-8e65-512974f66c58 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:40.750609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22db80f-c2e3-4a4c-8d39-6e98e278b7b3 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:40.785804 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6cb0aa-d76c-47c4-b083-d7135ff31e12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:40.786362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6cb0aa-d76c-47c4-b083-d7135ff31e12 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:40.786636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6cb0aa-d76c-47c4-b083-d7135ff31e12 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:40.786904 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6cb0aa-d76c-47c4-b083-d7135ff31e12 None admin] PATH_INFO: `/v3/roles/member` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:40.798080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6cb0aa-d76c-47c4-b083-d7135ff31e12 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:40.802830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-4d6cb0aa-d76c-47c4-b083-d7135ff31e12 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:40.802830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:40.802830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:40.802830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:40.802830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:40.802830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:40.802830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:40.802830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:40.802830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:40.802830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:40.802830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:40.802830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:40.802830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:40.802830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:40.802830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:40.802830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:40.803934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:40.803934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:40.803934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:40.803934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:40.803934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:40.803934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:40.803934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:40.803934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:40.803934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:40.803934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:40.803934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:40.803934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:40.803934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:40.803934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:40.803934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:40.803934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:40.804708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:40.804708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:40.804708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:40.804708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:40.804708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:40.804708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:40.804708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:40.804708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:40.804708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:40.804708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:40.804708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:40.804708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:40.804708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 107/212] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:40 2022] GET /identity/v3/roles/member => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:43:40.805609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc63d05-adf9-49f9-9eb4-d3943d864514 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:40.805870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc63d05-adf9-49f9-9eb4-d3943d864514 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:40.806145 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc63d05-adf9-49f9-9eb4-d3943d864514 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:40.823375 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-ecc63d05-adf9-49f9-9eb4-d3943d864514 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:40.824588 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ecc63d05-adf9-49f9-9eb4-d3943d864514 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:40.848394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92da738b-34a7-4c18-8e65-512974f66c58 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:40.875868 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:40.882487 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f321d8f-9294-4b19-940d-94653f3aa085 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:40.883093 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f321d8f-9294-4b19-940d-94653f3aa085 None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:40.883372 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f321d8f-9294-4b19-940d-94653f3aa085 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:40.883680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f321d8f-9294-4b19-940d-94653f3aa085 None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:40.884331 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f321d8f-9294-4b19-940d-94653f3aa085 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:40.885769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f321d8f-9294-4b19-940d-94653f3aa085 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:40.893124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 108/213] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:40.894135 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc63d05-adf9-49f9-9eb4-d3943d864514 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:40.897001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:40.919582 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 106/214] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:43:40 2022] POST /identity/v3/users => generated 293 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:43:41.010827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc63d05-adf9-49f9-9eb4-d3943d864514 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:41.049615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48cc8996-4953-4a93-93d6-2b8645af995a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:41.050164 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cc8996-4953-4a93-93d6-2b8645af995a None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:41.050448 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cc8996-4953-4a93-93d6-2b8645af995a None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:41.050724 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cc8996-4953-4a93-93d6-2b8645af995a None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:41.051410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48cc8996-4953-4a93-93d6-2b8645af995a None admin] RBAC: Adding query filter params (name=member) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:41.051748 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48cc8996-4953-4a93-93d6-2b8645af995a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:41.053167 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48cc8996-4953-4a93-93d6-2b8645af995a None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:41.063662 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 109/215] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:40 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:41.069262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f321d8f-9294-4b19-940d-94653f3aa085 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:41.168709 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48cc8996-4953-4a93-93d6-2b8645af995a None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:41.183270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f321d8f-9294-4b19-940d-94653f3aa085 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:41.225632 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93b7bfc-8983-475f-9895-8c4c8f7c183b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:41.226286 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93b7bfc-8983-475f-9895-8c4c8f7c183b None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:41.226568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93b7bfc-8983-475f-9895-8c4c8f7c183b None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:41.226847 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93b7bfc-8983-475f-9895-8c4c8f7c183b None admin] PATH_INFO: `/v3/users/efe74e40fff542f78c87cc1e602ef6e6` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:41.242524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93b7bfc-8983-475f-9895-8c4c8f7c183b None admin] RBAC: Authorizing `identity:get_user(user_id=efe74e40fff542f78c87cc1e602ef6e6)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:41.243414 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:41.244850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93b7bfc-8983-475f-9895-8c4c8f7c183b None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:41.246747 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 107/216] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:41 2022] GET /identity/v3/users/efe74e40fff542f78c87cc1e602ef6e6 => generated 293 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:41.253675 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93b7bfc-8983-475f-9895-8c4c8f7c183b None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:41.309352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48cc8996-4953-4a93-93d6-2b8645af995a None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:41.323324 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:41.353479 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0f4bd5-1cc9-43cb-b65f-522935ec19e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:41.354119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0f4bd5-1cc9-43cb-b65f-522935ec19e5 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:41.354738 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0f4bd5-1cc9-43cb-b65f-522935ec19e5 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:41.354853 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0f4bd5-1cc9-43cb-b65f-522935ec19e5 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:41.368053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0f4bd5-1cc9-43cb-b65f-522935ec19e5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:41.373998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-8d0f4bd5-1cc9-43cb-b65f-522935ec19e5 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 31 14:43:41.373998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:41.373998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:41.373998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:41.373998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:41.373998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:41.373998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:41.373998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:41.373998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:41.373998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:41.373998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:41.373998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:41.373998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:41.373998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:41.373998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:41.373998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:41.376022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:41.376022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:41.376022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:41.376022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:41.376022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:41.376022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:41.376022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:41.376022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:41.376022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:41.376022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:41.376022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:41.376022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:41.376022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:41.376022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:41.376022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:41.376022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:41.376809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:41.376809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:41.376809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:41.376809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:41.376809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:41.376809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:41.376809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:41.376809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:41.376809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:41.376809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:41.376809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 31 14:43:41.376809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:41.377409 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93b7bfc-8983-475f-9895-8c4c8f7c183b None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:41.377481 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 110/217] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:41 2022] GET /identity/v3/roles/reader => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:43:41.385875 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0f4bd5-1cc9-43cb-b65f-522935ec19e5 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:41.412727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bb29d6-e6d3-444d-932d-0360019cebf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:41.413479 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bb29d6-e6d3-444d-932d-0360019cebf2 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:41.413850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bb29d6-e6d3-444d-932d-0360019cebf2 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:41.414213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bb29d6-e6d3-444d-932d-0360019cebf2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:41.415038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36bb29d6-e6d3-444d-932d-0360019cebf2 None admin] RBAC: Adding query filter params (role.id=bb354a7b796d467d91ff4b184011796e, user.id=efe74e40fff542f78c87cc1e602ef6e6, scope.system=all) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:41.415480 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36bb29d6-e6d3-444d-932d-0360019cebf2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:41.417039 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36bb29d6-e6d3-444d-932d-0360019cebf2 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:41.423405 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 108/218] 10.222.0.36 () {56 vars in 1308 bytes} [Mon Oct 31 14:43:41 2022] GET /identity/v3/role_assignments?user.id=efe74e40fff542f78c87cc1e602ef6e6&scope.system=all&role.id=bb354a7b796d467d91ff4b184011796e => generated 226 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:41.514163 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0f4bd5-1cc9-43cb-b65f-522935ec19e5 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:41.526140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:41.555056 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab69115-69cd-4a7e-957b-8e9c9f681180 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:41.555728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab69115-69cd-4a7e-957b-8e9c9f681180 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:41.556004 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab69115-69cd-4a7e-957b-8e9c9f681180 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:41.556278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab69115-69cd-4a7e-957b-8e9c9f681180 None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:41.557037 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab69115-69cd-4a7e-957b-8e9c9f681180 None admin] RBAC: Adding query filter params (name=reader) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:41.557346 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab69115-69cd-4a7e-957b-8e9c9f681180 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:41.558783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab69115-69cd-4a7e-957b-8e9c9f681180 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:41.569059 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 111/219] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:41 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:41.575045 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bb29d6-e6d3-444d-932d-0360019cebf2 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:41.686183 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bb29d6-e6d3-444d-932d-0360019cebf2 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:41.705114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:41.723084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3669117c-e5b1-473c-832d-92be2643f94a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:41.723748 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3669117c-e5b1-473c-832d-92be2643f94a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:41.724029 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3669117c-e5b1-473c-832d-92be2643f94a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:41.724301 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3669117c-e5b1-473c-832d-92be2643f94a None admin] PATH_INFO: `/v3/users/e780cb5a88c341ada14aa47d4b51353e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:41.725442 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3669117c-e5b1-473c-832d-92be2643f94a None admin] RBAC: Authorizing `identity:get_user(user_id=e780cb5a88c341ada14aa47d4b51353e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:41.727060 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3669117c-e5b1-473c-832d-92be2643f94a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:41.733311 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 109/220] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:41 2022] GET /identity/v3/users/e780cb5a88c341ada14aa47d4b51353e => generated 297 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:41.735458 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab69115-69cd-4a7e-957b-8e9c9f681180 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:41.840656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab69115-69cd-4a7e-957b-8e9c9f681180 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:41.877128 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ed68de-0723-45ae-a14c-3db169294e28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:41.877743 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ed68de-0723-45ae-a14c-3db169294e28 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:41.878024 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ed68de-0723-45ae-a14c-3db169294e28 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:41.878296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ed68de-0723-45ae-a14c-3db169294e28 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:41.928379 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:41.952954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ed68de-0723-45ae-a14c-3db169294e28 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:41.954519 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ed68de-0723-45ae-a14c-3db169294e28 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:42.063694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 112/221] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:43:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 31 14:43:42.066316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3669117c-e5b1-473c-832d-92be2643f94a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:42.175310 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3669117c-e5b1-473c-832d-92be2643f94a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:42.210932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3297e33c-746c-4ce7-9b65-d38d9a083bf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:42.211577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3297e33c-746c-4ce7-9b65-d38d9a083bf1 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:42.211866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3297e33c-746c-4ce7-9b65-d38d9a083bf1 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:42.212137 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3297e33c-746c-4ce7-9b65-d38d9a083bf1 None admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:42.213262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3297e33c-746c-4ce7-9b65-d38d9a083bf1 None admin] RBAC: Authorizing `identity:get_project(project_id=5847498d7fa84446a3389fbb99498368)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:42.214728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3297e33c-746c-4ce7-9b65-d38d9a083bf1 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:42.216460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 110/222] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:42 2022] GET /identity/v3/projects/5847498d7fa84446a3389fbb99498368 => generated 302 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:42.223597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ed68de-0723-45ae-a14c-3db169294e28 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:42.294377 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:42.326033 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ed68de-0723-45ae-a14c-3db169294e28 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:42.364407 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7447e9ed-ebaf-49e4-8d52-79046e7cd86e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:42.365023 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7447e9ed-ebaf-49e4-8d52-79046e7cd86e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:42.365399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7447e9ed-ebaf-49e4-8d52-79046e7cd86e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:42.365655 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7447e9ed-ebaf-49e4-8d52-79046e7cd86e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:42.366376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7447e9ed-ebaf-49e4-8d52-79046e7cd86e None admin] RBAC: Adding query filter params (role.id=d5174defb0fa4869819ae433429cb2df, scope.project.id=5847498d7fa84446a3389fbb99498368, user.id=e780cb5a88c341ada14aa47d4b51353e) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:42.366694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7447e9ed-ebaf-49e4-8d52-79046e7cd86e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:42.368191 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7447e9ed-ebaf-49e4-8d52-79046e7cd86e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:42.379722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 113/223] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:43:42 2022] GET /identity/v3/role_assignments?user.id=e780cb5a88c341ada14aa47d4b51353e&scope.project.id=5847498d7fa84446a3389fbb99498368&role.id=d5174defb0fa4869819ae433429cb2df => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:43.291781 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db3f4cb-ac3f-4a14-99c5-ee0b9a3893c1 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:43.292295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db3f4cb-ac3f-4a14-99c5-ee0b9a3893c1 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:43.292295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db3f4cb-ac3f-4a14-99c5-ee0b9a3893c1 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:43.293985 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 111/224] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:43.299789 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce1f46c-9273-47d5-9d27-0019cb641a42 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:43.300223 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce1f46c-9273-47d5-9d27-0019cb641a42 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:43.300396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce1f46c-9273-47d5-9d27-0019cb641a42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:43.319060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-4ce1f46c-9273-47d5-9d27-0019cb641a42 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:43.320274 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-4ce1f46c-9273-47d5-9d27-0019cb641a42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:43.392434 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 114/225] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:43.457467 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3916b5be-8f80-402e-b1f5-ffe9414fe947 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:43.457777 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3916b5be-8f80-402e-b1f5-ffe9414fe947 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:43.458027 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3916b5be-8f80-402e-b1f5-ffe9414fe947 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:43.459526 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 112/226] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:43.464578 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a70443-cb1d-4bf6-ba9a-45ec53d00301 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:43.464870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a70443-cb1d-4bf6-ba9a-45ec53d00301 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:43.465149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a70443-cb1d-4bf6-ba9a-45ec53d00301 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:43.485267 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-21a70443-cb1d-4bf6-ba9a-45ec53d00301 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:43.486474 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-21a70443-cb1d-4bf6-ba9a-45ec53d00301 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:43.555031 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:43.565057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 115/227] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:43.637409 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8211769c-1afe-4872-922b-b24de58d16c3 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:43.637728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8211769c-1afe-4872-922b-b24de58d16c3 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:43.637966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8211769c-1afe-4872-922b-b24de58d16c3 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:43.639440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 113/228] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:43.644552 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31028dcb-434e-41da-a99d-15c2cbece7ea None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:43.644839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31028dcb-434e-41da-a99d-15c2cbece7ea None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:43.645112 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31028dcb-434e-41da-a99d-15c2cbece7ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:43.663543 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-31028dcb-434e-41da-a99d-15c2cbece7ea None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:43.664739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-31028dcb-434e-41da-a99d-15c2cbece7ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:43.670977 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8211769c-1afe-4872-922b-b24de58d16c3 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:43.731404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:43.732605 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 116/229] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:43.778937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8211769c-1afe-4872-922b-b24de58d16c3 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:43.813348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf89a97-e79e-46f8-bdc7-6ee19f327c2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:43.813903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf89a97-e79e-46f8-bdc7-6ee19f327c2f None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:43.814173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf89a97-e79e-46f8-bdc7-6ee19f327c2f None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:43.814444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf89a97-e79e-46f8-bdc7-6ee19f327c2f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:43.824521 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf89a97-e79e-46f8-bdc7-6ee19f327c2f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:43.830064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-cdf89a97-e79e-46f8-bdc7-6ee19f327c2f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 31 14:43:43.830064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:43.830064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:43.830064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:43.830064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:43.830064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:43.830064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:43.830064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:43.830064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:43.830064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:43.830064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:43.830064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:43.830064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:43.830064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:43.830064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:43.830064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:43.830913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:43.830913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:43.830913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:43.830913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:43.830913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:43.830913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:43.830913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:43.830913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:43.830913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:43.830913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:43.830913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:43.830913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:43.830913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:43.830913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:43.830913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:43.830913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:43.831737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:43.831737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:43.831737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:43.831737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:43.831737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:43.831737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:43.831737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:43.831737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:43.831737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:43.831737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:43.831737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 31 14:43:43.831737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:43.831737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 114/230] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:43 2022] GET /identity/v3/roles/reader => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:43.837072 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31028dcb-434e-41da-a99d-15c2cbece7ea None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:43.847105 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf89a97-e79e-46f8-bdc7-6ee19f327c2f None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:43.941178 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31028dcb-434e-41da-a99d-15c2cbece7ea None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:43.955749 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf89a97-e79e-46f8-bdc7-6ee19f327c2f None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:43.967216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:43.971085 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:43.979730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4180e687-9f03-46e0-818b-a98eea20ab43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:43.980344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4180e687-9f03-46e0-818b-a98eea20ab43 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:43.980650 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4180e687-9f03-46e0-818b-a98eea20ab43 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:43.980891 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4180e687-9f03-46e0-818b-a98eea20ab43 None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:43.981555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4180e687-9f03-46e0-818b-a98eea20ab43 None admin] RBAC: Adding query filter params (name=reader) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:43.981860 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4180e687-9f03-46e0-818b-a98eea20ab43 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:43.983222 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4180e687-9f03-46e0-818b-a98eea20ab43 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:43.992475 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 117/231] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:43 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:43.997485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3835017e-145d-43d9-ba27-f3b097be59ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:43.997843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4180e687-9f03-46e0-818b-a98eea20ab43 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:43.998130 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3835017e-145d-43d9-ba27-f3b097be59ae None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:43.998407 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3835017e-145d-43d9-ba27-f3b097be59ae None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:43.998749 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3835017e-145d-43d9-ba27-f3b097be59ae None admin] PATH_INFO: `/v3/domains/default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:44.000396 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3835017e-145d-43d9-ba27-f3b097be59ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:44.001832 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3835017e-145d-43d9-ba27-f3b097be59ae None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:44.003420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 115/232] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:43:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:44.004217 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3835017e-145d-43d9-ba27-f3b097be59ae None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:44.097622 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4180e687-9f03-46e0-818b-a98eea20ab43 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:44.101787 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3835017e-145d-43d9-ba27-f3b097be59ae None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:44.131402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7709421-746d-4bcf-af03-f1734864707b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:44.132049 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7709421-746d-4bcf-af03-f1734864707b None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:44.132330 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7709421-746d-4bcf-af03-f1734864707b None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:44.132600 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7709421-746d-4bcf-af03-f1734864707b None admin] PATH_INFO: `/v3/users/d0ebd64a902d42f3bf9133cccfc71c2e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:44.136077 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e86920f-d7ad-41a2-bc50-8f7ae4257769 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:44.136692 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e86920f-d7ad-41a2-bc50-8f7ae4257769 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:44.136969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e86920f-d7ad-41a2-bc50-8f7ae4257769 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:44.137240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e86920f-d7ad-41a2-bc50-8f7ae4257769 None admin] PATH_INFO: `/v3/roles/member` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:44.145468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7709421-746d-4bcf-af03-f1734864707b None admin] RBAC: Authorizing `identity:get_user(user_id=d0ebd64a902d42f3bf9133cccfc71c2e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:44.146940 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7709421-746d-4bcf-af03-f1734864707b None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:44.147306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e86920f-d7ad-41a2-bc50-8f7ae4257769 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:44.148597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 118/233] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:43 2022] GET /identity/v3/users/d0ebd64a902d42f3bf9133cccfc71c2e => generated 293 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:44.149341 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7709421-746d-4bcf-af03-f1734864707b None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:44.152641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-8e86920f-d7ad-41a2-bc50-8f7ae4257769 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:44.152641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:44.152641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:44.152641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:44.152641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:44.152641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:44.152641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:44.152641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:44.152641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:44.152641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:44.152641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:44.152641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:44.152641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:44.152641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:44.152641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:44.152641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:44.154284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:44.154284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:44.154284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:44.154284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:44.154284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:44.154284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:44.154284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:44.154284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:44.154284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:44.154284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:44.154284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:44.154284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:44.154284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:44.154284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:44.154284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:44.154284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:44.155430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:44.155430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:44.155430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:44.155430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:44.155430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:44.155430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:44.155430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:44.155430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:44.155430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:44.155430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:44.155430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:44.155430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:44.155430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 116/234] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:44.155430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e86920f-d7ad-41a2-bc50-8f7ae4257769 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:44.198242 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:44.201271 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:44.253402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7709421-746d-4bcf-af03-f1734864707b None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:44.254370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e86920f-d7ad-41a2-bc50-8f7ae4257769 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:44.286159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5326ed44-7ca2-470f-bad3-46f085414aa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:44.286752 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5326ed44-7ca2-470f-bad3-46f085414aa1 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:44.287028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5326ed44-7ca2-470f-bad3-46f085414aa1 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:44.287296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5326ed44-7ca2-470f-bad3-46f085414aa1 None admin] PATH_INFO: `/v3/groups` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:44.287572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b019d5-f426-42d8-b3f1-2cc0f6586207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:44.287948 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5326ed44-7ca2-470f-bad3-46f085414aa1 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:44.288173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b019d5-f426-42d8-b3f1-2cc0f6586207 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:44.288452 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b019d5-f426-42d8-b3f1-2cc0f6586207 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:44.288720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b019d5-f426-42d8-b3f1-2cc0f6586207 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:44.289362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5326ed44-7ca2-470f-bad3-46f085414aa1 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:44.289455 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b019d5-f426-42d8-b3f1-2cc0f6586207 None admin] RBAC: Adding query filter params (role.id=d5174defb0fa4869819ae433429cb2df, user.id=d0ebd64a902d42f3bf9133cccfc71c2e, scope.system=all) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:44.289729 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b019d5-f426-42d8-b3f1-2cc0f6586207 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:44.291159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b019d5-f426-42d8-b3f1-2cc0f6586207 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:44.296672 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 117/235] 10.222.0.36 () {56 vars in 1308 bytes} [Mon Oct 31 14:43:44 2022] GET /identity/v3/role_assignments?user.id=d0ebd64a902d42f3bf9133cccfc71c2e&scope.system=all&role.id=d5174defb0fa4869819ae433429cb2df => generated 226 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:44.297663 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b019d5-f426-42d8-b3f1-2cc0f6586207 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:44.324194 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 119/236] 10.222.0.36 () {60 vars in 1134 bytes} [Mon Oct 31 14:43:44 2022] POST /identity/v3/groups => generated 229 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:43:44.423220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b019d5-f426-42d8-b3f1-2cc0f6586207 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:44.433114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:44.465490 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65107f82-aa1a-4895-ac5b-3253e968520b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:44.466125 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65107f82-aa1a-4895-ac5b-3253e968520b None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:44.466404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65107f82-aa1a-4895-ac5b-3253e968520b None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:44.466674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65107f82-aa1a-4895-ac5b-3253e968520b None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:44.467368 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65107f82-aa1a-4895-ac5b-3253e968520b None admin] RBAC: Adding query filter params (name=member) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:44.467731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65107f82-aa1a-4895-ac5b-3253e968520b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:44.469277 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65107f82-aa1a-4895-ac5b-3253e968520b None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:44.479497 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 118/237] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:44.485411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5326ed44-7ca2-470f-bad3-46f085414aa1 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:44.600807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5326ed44-7ca2-470f-bad3-46f085414aa1 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:44.604297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:44.638793 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13f468c-daed-47c6-9391-22c36c5892fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:44.639431 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13f468c-daed-47c6-9391-22c36c5892fb None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:44.640275 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13f468c-daed-47c6-9391-22c36c5892fb None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:44.640928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13f468c-daed-47c6-9391-22c36c5892fb None admin] PATH_INFO: `/v3/users/efe74e40fff542f78c87cc1e602ef6e6` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:44.641749 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13f468c-daed-47c6-9391-22c36c5892fb None admin] RBAC: Authorizing `identity:get_user(user_id=efe74e40fff542f78c87cc1e602ef6e6)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:44.643185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13f468c-daed-47c6-9391-22c36c5892fb None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:44.644980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 120/238] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:44 2022] GET /identity/v3/users/efe74e40fff542f78c87cc1e602ef6e6 => generated 293 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:44.650421 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65107f82-aa1a-4895-ac5b-3253e968520b None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:44.752866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65107f82-aa1a-4895-ac5b-3253e968520b None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:44.787853 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a10e25a-5925-42aa-aa84-bbca1753b3b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:44.788508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a10e25a-5925-42aa-aa84-bbca1753b3b3 None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:44.788789 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a10e25a-5925-42aa-aa84-bbca1753b3b3 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:44.789062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a10e25a-5925-42aa-aa84-bbca1753b3b3 None admin] PATH_INFO: `/v3/system/users/efe74e40fff542f78c87cc1e602ef6e6/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:44.790476 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a10e25a-5925-42aa-aa84-bbca1753b3b3 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=efe74e40fff542f78c87cc1e602ef6e6, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:44.791877 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a10e25a-5925-42aa-aa84-bbca1753b3b3 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:44.800915 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 119/239] 10.222.0.36 () {58 vars in 1334 bytes} [Mon Oct 31 14:43:44 2022] PUT /identity/v3/system/users/efe74e40fff542f78c87cc1e602ef6e6/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:43:46.486961 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26984a03-bab0-4b87-8318-d3b5dd877875 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:46.487478 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26984a03-bab0-4b87-8318-d3b5dd877875 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:46.487577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26984a03-bab0-4b87-8318-d3b5dd877875 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:46.488916 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 121/240] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:46.494232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7ed9d3-7267-4456-89d0-f88eb07fea28 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:46.494651 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7ed9d3-7267-4456-89d0-f88eb07fea28 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:46.494817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7ed9d3-7267-4456-89d0-f88eb07fea28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:46.513991 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-3e7ed9d3-7267-4456-89d0-f88eb07fea28 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:46.516338 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-3e7ed9d3-7267-4456-89d0-f88eb07fea28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:46.548424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:46.565435 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51f72a9-6b91-4635-bfea-44c9ea86437e None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:46.565717 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51f72a9-6b91-4635-bfea-44c9ea86437e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:46.566158 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51f72a9-6b91-4635-bfea-44c9ea86437e None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:46.567500 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 122/241] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:46.572777 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4fe6e5-f3aa-413b-9e2d-ad1e684517d0 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:46.573063 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4fe6e5-f3aa-413b-9e2d-ad1e684517d0 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:46.573332 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4fe6e5-f3aa-413b-9e2d-ad1e684517d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:46.592316 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-1e4fe6e5-f3aa-413b-9e2d-ad1e684517d0 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:46.594272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1e4fe6e5-f3aa-413b-9e2d-ad1e684517d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:46.595981 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 120/242] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:46.663489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 123/243] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:43:46.864787 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7ed9d3-7267-4456-89d0-f88eb07fea28 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:46.938105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4fe6e5-f3aa-413b-9e2d-ad1e684517d0 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:46.963870 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7ed9d3-7267-4456-89d0-f88eb07fea28 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:46.997822 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca00021-42df-40d2-ac7a-120ead909b75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:46.998440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca00021-42df-40d2-ac7a-120ead909b75 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:46.998716 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca00021-42df-40d2-ac7a-120ead909b75 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:46.999570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca00021-42df-40d2-ac7a-120ead909b75 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:47.009304 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca00021-42df-40d2-ac7a-120ead909b75 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:47.014084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-cca00021-42df-40d2-ac7a-120ead909b75 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 31 14:43:47.014084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:47.014084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:47.014084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:47.014084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:47.014084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:47.014084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:47.014084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:47.014084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:47.014084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:47.014084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:47.014084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:47.014084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:47.014084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:47.014084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:47.014084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:47.015243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:47.015243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:47.015243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:47.015243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:47.015243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:47.015243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:47.015243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:47.015243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:47.015243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:47.015243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:47.015243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:47.015243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:47.015243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:47.015243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:47.015243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:47.015243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:47.016065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:47.016065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:47.016065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:47.016065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:47.016065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:47.016065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:47.016065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:47.016065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:47.016065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:47.016065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:47.016065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 31 14:43:47.016065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:47.016065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 121/244] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:46 2022] GET /identity/v3/roles/reader => generated 84 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:47.017930 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb548b3d-e0f6-4734-a471-045f0e910e8e None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:47.018210 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb548b3d-e0f6-4734-a471-045f0e910e8e None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:47.018483 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb548b3d-e0f6-4734-a471-045f0e910e8e None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:47.024002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 122/245] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:47 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:47.024796 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb548b3d-e0f6-4734-a471-045f0e910e8e None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:47.040004 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4fe6e5-f3aa-413b-9e2d-ad1e684517d0 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:47.041498 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:47.049857 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:47.075140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6b3128-ec93-4f6a-bacc-9a7c8fcbeba2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:47.075801 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6b3128-ec93-4f6a-bacc-9a7c8fcbeba2 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:47.076079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6b3128-ec93-4f6a-bacc-9a7c8fcbeba2 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:47.076352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6b3128-ec93-4f6a-bacc-9a7c8fcbeba2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:47.078051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6b3128-ec93-4f6a-bacc-9a7c8fcbeba2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:47.079546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6b3128-ec93-4f6a-bacc-9a7c8fcbeba2 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:47.081137 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 124/246] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:43:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:47.082235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a8abfb-31ab-47d8-9b44-d1d67d561c31 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:47.082502 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a8abfb-31ab-47d8-9b44-d1d67d561c31 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:47.082774 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a8abfb-31ab-47d8-9b44-d1d67d561c31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:47.100846 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-98a8abfb-31ab-47d8-9b44-d1d67d561c31 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:47.102016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-98a8abfb-31ab-47d8-9b44-d1d67d561c31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:47.127608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb548b3d-e0f6-4734-a471-045f0e910e8e None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:47.161669 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f1445e-b948-46e1-8a64-0009b20ed78c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:47.162269 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f1445e-b948-46e1-8a64-0009b20ed78c None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:47.162544 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f1445e-b948-46e1-8a64-0009b20ed78c None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:47.162813 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f1445e-b948-46e1-8a64-0009b20ed78c None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:47.163470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f1445e-b948-46e1-8a64-0009b20ed78c None admin] RBAC: Adding query filter params (name=reader) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:47.163806 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f1445e-b948-46e1-8a64-0009b20ed78c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:47.165168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f1445e-b948-46e1-8a64-0009b20ed78c None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:47.167931 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 125/247] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:47.168779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a8abfb-31ab-47d8-9b44-d1d67d561c31 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:47.183585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 123/248] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:47 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:47.189089 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f1445e-b948-46e1-8a64-0009b20ed78c None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:47.275880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:47.279275 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:47.279429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a8abfb-31ab-47d8-9b44-d1d67d561c31 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:47.297646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f1445e-b948-46e1-8a64-0009b20ed78c None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:47.316317 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756b47ef-9af4-41ad-bd71-00a5213c931a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:47.316914 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756b47ef-9af4-41ad-bd71-00a5213c931a None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:47.317187 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756b47ef-9af4-41ad-bd71-00a5213c931a None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:47.317454 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756b47ef-9af4-41ad-bd71-00a5213c931a None admin] PATH_INFO: `/v3/groups` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:47.318082 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756b47ef-9af4-41ad-bd71-00a5213c931a None admin] RBAC: Authorizing `identity:create_group()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:47.319951 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756b47ef-9af4-41ad-bd71-00a5213c931a None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:47.334632 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0dcc93c-d95c-44a1-bc48-5a8f6588d226 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:47.335306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dcc93c-d95c-44a1-bc48-5a8f6588d226 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:47.335616 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dcc93c-d95c-44a1-bc48-5a8f6588d226 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:47.335901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dcc93c-d95c-44a1-bc48-5a8f6588d226 None admin] PATH_INFO: `/v3/users/d0ebd64a902d42f3bf9133cccfc71c2e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:47.336298 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 126/249] 10.222.0.36 () {60 vars in 1134 bytes} [Mon Oct 31 14:43:47 2022] POST /identity/v3/groups => generated 226 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:43:47.340971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0dcc93c-d95c-44a1-bc48-5a8f6588d226 None admin] RBAC: Authorizing `identity:get_user(user_id=d0ebd64a902d42f3bf9133cccfc71c2e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:47.342683 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0dcc93c-d95c-44a1-bc48-5a8f6588d226 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:47.344841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 124/250] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:47 2022] GET /identity/v3/users/d0ebd64a902d42f3bf9133cccfc71c2e => generated 293 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:47.350873 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756b47ef-9af4-41ad-bd71-00a5213c931a None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:47.445249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0dcc93c-d95c-44a1-bc48-5a8f6588d226 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:47.466743 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756b47ef-9af4-41ad-bd71-00a5213c931a None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:47.510955 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57d01c4-eee0-4258-af99-a89f0828a1a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:47.511956 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57d01c4-eee0-4258-af99-a89f0828a1a2 None admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:47.512234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57d01c4-eee0-4258-af99-a89f0828a1a2 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:47.512509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57d01c4-eee0-4258-af99-a89f0828a1a2 None admin] PATH_INFO: `/v3/system/users/d0ebd64a902d42f3bf9133cccfc71c2e/roles/d5174defb0fa4869819ae433429cb2df` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:47.513974 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57d01c4-eee0-4258-af99-a89f0828a1a2 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=d0ebd64a902d42f3bf9133cccfc71c2e, role_id=d5174defb0fa4869819ae433429cb2df)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:47.516759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57d01c4-eee0-4258-af99-a89f0828a1a2 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:47.528542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 127/251] 10.222.0.36 () {58 vars in 1334 bytes} [Mon Oct 31 14:43:47 2022] PUT /identity/v3/system/users/d0ebd64a902d42f3bf9133cccfc71c2e/roles/d5174defb0fa4869819ae433429cb2df => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:43:47.575758 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0dcc93c-d95c-44a1-bc48-5a8f6588d226 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:47.614022 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2822fba-02e9-482f-86ee-d7dadd2965d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:47.614719 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2822fba-02e9-482f-86ee-d7dadd2965d5 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:47.615002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2822fba-02e9-482f-86ee-d7dadd2965d5 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:47.615277 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2822fba-02e9-482f-86ee-d7dadd2965d5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:47.629805 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2822fba-02e9-482f-86ee-d7dadd2965d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:47.635397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-d2822fba-02e9-482f-86ee-d7dadd2965d5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:47.635397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:47.635397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:47.635397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:47.635397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:47.635397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:47.635397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:47.635397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:47.635397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:47.635397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:47.635397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:47.635397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:47.635397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:47.635397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:47.635397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:47.635397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:47.636376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:47.636376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:47.636376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:47.636376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:47.636376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:47.636376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:47.636376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:47.636376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:47.636376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:47.636376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:47.636376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:47.636376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:47.636376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:47.636376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:47.636376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:47.636376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:47.637149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:47.637149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:47.637149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:47.637149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:47.637149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:47.637149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:47.637149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:47.637149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:47.637149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:47.637149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:47.637149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:47.637149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:47.637149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 125/252] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:47.643286 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57d01c4-eee0-4258-af99-a89f0828a1a2 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:47.677170 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:47.765492 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57d01c4-eee0-4258-af99-a89f0828a1a2 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:47.802090 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8502e63-46d8-4ca4-9291-1ae7ecd9673c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:47.802717 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8502e63-46d8-4ca4-9291-1ae7ecd9673c None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:47.803000 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8502e63-46d8-4ca4-9291-1ae7ecd9673c None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:47.803274 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8502e63-46d8-4ca4-9291-1ae7ecd9673c None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:47.803980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8502e63-46d8-4ca4-9291-1ae7ecd9673c None admin] RBAC: Adding query filter params (name=member) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:47.804288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8502e63-46d8-4ca4-9291-1ae7ecd9673c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:47.805676 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8502e63-46d8-4ca4-9291-1ae7ecd9673c None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:47.815496 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 128/253] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:47.821312 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2822fba-02e9-482f-86ee-d7dadd2965d5 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:47.845066 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:47.950951 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2822fba-02e9-482f-86ee-d7dadd2965d5 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:47.989395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09992eb0-6f72-4b57-a981-91917a852328 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:47.990042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09992eb0-6f72-4b57-a981-91917a852328 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:47.990318 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09992eb0-6f72-4b57-a981-91917a852328 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:47.990592 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09992eb0-6f72-4b57-a981-91917a852328 None admin] PATH_INFO: `/v3/users/efe74e40fff542f78c87cc1e602ef6e6` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:47.993619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09992eb0-6f72-4b57-a981-91917a852328 None admin] RBAC: Authorizing `identity:get_user(user_id=efe74e40fff542f78c87cc1e602ef6e6)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:47.995102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09992eb0-6f72-4b57-a981-91917a852328 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:47.996964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 126/254] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:47 2022] GET /identity/v3/users/efe74e40fff542f78c87cc1e602ef6e6 => generated 293 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:48.002995 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8502e63-46d8-4ca4-9291-1ae7ecd9673c None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:48.109749 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:48.130209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8502e63-46d8-4ca4-9291-1ae7ecd9673c None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:48.168237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd2dcc7-74fb-4cc4-9dc6-176a06e9e7de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:48.168850 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd2dcc7-74fb-4cc4-9dc6-176a06e9e7de None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:48.169132 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd2dcc7-74fb-4cc4-9dc6-176a06e9e7de None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:48.169404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd2dcc7-74fb-4cc4-9dc6-176a06e9e7de None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:48.170135 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd2dcc7-74fb-4cc4-9dc6-176a06e9e7de None admin] RBAC: Adding query filter params (role.id=bb354a7b796d467d91ff4b184011796e, user.id=efe74e40fff542f78c87cc1e602ef6e6, scope.system=all) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:48.170443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd2dcc7-74fb-4cc4-9dc6-176a06e9e7de None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:48.172395 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd2dcc7-74fb-4cc4-9dc6-176a06e9e7de None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:48.186990 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 129/255] 10.222.0.36 () {56 vars in 1308 bytes} [Mon Oct 31 14:43:48 2022] GET /identity/v3/role_assignments?user.id=efe74e40fff542f78c87cc1e602ef6e6&scope.system=all&role.id=bb354a7b796d467d91ff4b184011796e => generated 510 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:50.138224 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e70a113-2b48-4afc-ab89-39d0a101856b None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:50.139176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e70a113-2b48-4afc-ab89-39d0a101856b None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:50.140323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a6a33b-8b94-4bda-b6d7-10f4ace7d797 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:50.140842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a6a33b-8b94-4bda-b6d7-10f4ace7d797 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:50.141073 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a6a33b-8b94-4bda-b6d7-10f4ace7d797 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:50.141639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e70a113-2b48-4afc-ab89-39d0a101856b None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:50.142367 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 130/256] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:43:50.143544 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 127/257] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:50 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:50.148110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d432451-6c4e-4d36-ad46-0c860c7e023a None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:50.148396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d432451-6c4e-4d36-ad46-0c860c7e023a None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:50.148673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d432451-6c4e-4d36-ad46-0c860c7e023a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:50.152956 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389dd7e6-d935-480e-b882-0b2bad33869c None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:50.153243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389dd7e6-d935-480e-b882-0b2bad33869c None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:50.153508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389dd7e6-d935-480e-b882-0b2bad33869c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:50.188640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-7d432451-6c4e-4d36-ad46-0c860c7e023a None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:50.189866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7d432451-6c4e-4d36-ad46-0c860c7e023a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:50.192994 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-389dd7e6-d935-480e-b882-0b2bad33869c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:50.194193 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-389dd7e6-d935-480e-b882-0b2bad33869c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:50.308269 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 128/258] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:50 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 31 14:43:50.308581 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 131/259] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:50.335571 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1593f1bd-77d3-4238-a148-e72680a94e03 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:50.335891 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1593f1bd-77d3-4238-a148-e72680a94e03 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:50.336130 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1593f1bd-77d3-4238-a148-e72680a94e03 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:50.337494 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 129/260] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:50.343393 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2720e255-ce2f-4030-a82c-0435c95058bc None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:50.343757 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2720e255-ce2f-4030-a82c-0435c95058bc None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:50.343983 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2720e255-ce2f-4030-a82c-0435c95058bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:50.368886 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-2720e255-ce2f-4030-a82c-0435c95058bc None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:50.370086 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2720e255-ce2f-4030-a82c-0435c95058bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:50.384155 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 132/261] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:50 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 31 14:43:50.736725 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1593f1bd-77d3-4238-a148-e72680a94e03 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:50.753772 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1593f1bd-77d3-4238-a148-e72680a94e03 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:50.758186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4d2c48-e6f0-4dce-84ec-a1e92e7ff126 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:50.758818 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4d2c48-e6f0-4dce-84ec-a1e92e7ff126 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:50.759096 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4d2c48-e6f0-4dce-84ec-a1e92e7ff126 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:50.759371 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4d2c48-e6f0-4dce-84ec-a1e92e7ff126 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:50.773619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4d2c48-e6f0-4dce-84ec-a1e92e7ff126 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:50.778518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-0f4d2c48-e6f0-4dce-84ec-a1e92e7ff126 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:43:50.778518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:50.778518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:50.778518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:50.778518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:50.778518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:50.778518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:50.778518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:50.778518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:50.778518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:50.778518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:50.778518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:50.778518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:50.778518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:50.778518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:50.778518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:50.779388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:50.779388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:50.779388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:50.779388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:50.779388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:50.779388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:50.779388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:50.779388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:50.779388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:50.779388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:50.779388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:50.779388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:50.779388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:50.779388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:50.779388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:50.779388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:50.780792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:50.780792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:50.780792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:50.780792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:50.780792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:50.780792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:50.780792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:50.780792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:50.780792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:50.780792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:50.780792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:43:50.780792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:50.780792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 130/262] 10.222.0.36 () {56 vars in 1135 bytes} [Mon Oct 31 14:43:50 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 44 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:50.786193 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2720e255-ce2f-4030-a82c-0435c95058bc None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:50.800815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2720e255-ce2f-4030-a82c-0435c95058bc None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:50.808152 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1682707-76fb-4e00-9003-1a51e9580218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:50.808765 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1682707-76fb-4e00-9003-1a51e9580218 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:50.809042 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1682707-76fb-4e00-9003-1a51e9580218 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:50.809313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1682707-76fb-4e00-9003-1a51e9580218 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:50.809979 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1682707-76fb-4e00-9003-1a51e9580218 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:50.810279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1682707-76fb-4e00-9003-1a51e9580218 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:50.812242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: WARNING py.warnings [None req-b1682707-76fb-4e00-9003-1a51e9580218 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 31 14:43:50.812242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: warnings.warn(msg) Oct 31 14:43:50.812242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: Oct 31 14:43:50.813158 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1682707-76fb-4e00-9003-1a51e9580218 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:50.822095 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4d2c48-e6f0-4dce-84ec-a1e92e7ff126 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:50.833203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 133/263] 10.222.0.36 () {56 vars in 1132 bytes} [Mon Oct 31 14:43:50 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:50.839698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1682707-76fb-4e00-9003-1a51e9580218 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:50.849459 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:50.857581 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1682707-76fb-4e00-9003-1a51e9580218 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:50.870536 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce09e0c-f9de-47e7-830b-ccdfc081d67f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:50.871205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce09e0c-f9de-47e7-830b-ccdfc081d67f admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:50.871482 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce09e0c-f9de-47e7-830b-ccdfc081d67f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:50.871782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce09e0c-f9de-47e7-830b-ccdfc081d67f admin admin] PATH_INFO: `/v3/groups/496de8c8aebb422fbcae2f9e32bba846` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:50.885384 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce09e0c-f9de-47e7-830b-ccdfc081d67f admin admin] RBAC: Authorizing `identity:get_group(group_id=496de8c8aebb422fbcae2f9e32bba846)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:50.886431 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: WARNING py.warnings [None req-cce09e0c-f9de-47e7-830b-ccdfc081d67f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 31 14:43:50.886431 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: warnings.warn(msg) Oct 31 14:43:50.886431 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: Oct 31 14:43:50.887695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce09e0c-f9de-47e7-830b-ccdfc081d67f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:50.894168 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 134/264] 10.222.0.36 () {56 vars in 1201 bytes} [Mon Oct 31 14:43:50 2022] GET /identity/v3/groups/496de8c8aebb422fbcae2f9e32bba846 => generated 226 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:50.895281 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce09e0c-f9de-47e7-830b-ccdfc081d67f admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:50.911166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce09e0c-f9de-47e7-830b-ccdfc081d67f admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:50.921519 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f92c5f-4bde-4250-b7e5-f0b51bfeb36e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:50.922172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f92c5f-4bde-4250-b7e5-f0b51bfeb36e admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:50.922461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f92c5f-4bde-4250-b7e5-f0b51bfeb36e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:50.922733 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f92c5f-4bde-4250-b7e5-f0b51bfeb36e admin admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:50.924454 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f92c5f-4bde-4250-b7e5-f0b51bfeb36e admin admin] RBAC: Authorizing `identity:get_project(project_id=dc35ec7707ee4a09b3784f8b3e76686c)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:50.926560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f92c5f-4bde-4250-b7e5-f0b51bfeb36e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:50.929739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 135/265] 10.222.0.36 () {56 vars in 1207 bytes} [Mon Oct 31 14:43:50 2022] GET /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c => generated 298 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:50.933240 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f92c5f-4bde-4250-b7e5-f0b51bfeb36e admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:50.946797 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f92c5f-4bde-4250-b7e5-f0b51bfeb36e admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:50.956926 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323d0697-ad97-46fd-9e9a-556dc917acab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:50.957558 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323d0697-ad97-46fd-9e9a-556dc917acab admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:50.957836 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323d0697-ad97-46fd-9e9a-556dc917acab admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:50.958107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323d0697-ad97-46fd-9e9a-556dc917acab admin admin] PATH_INFO: `/v3/roles/member` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:50.980707 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323d0697-ad97-46fd-9e9a-556dc917acab admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:50.986464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-323d0697-ad97-46fd-9e9a-556dc917acab admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:50.986464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:50.986464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:50.986464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:50.986464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:50.986464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:50.986464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:50.986464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:50.986464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:50.986464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:50.986464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:50.986464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:50.986464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:50.986464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:50.986464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:50.986464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:50.987331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:50.987331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:50.987331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:50.987331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:50.987331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:50.987331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:50.987331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:50.987331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:50.987331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:50.987331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:50.987331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:50.987331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:50.987331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:50.987331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:50.987331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:50.987331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:50.988142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:50.988142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:50.988142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:50.988142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:50.988142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:50.988142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:50.988142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:50.988142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:50.988142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:50.988142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:50.988142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:50.988142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:50.989188 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 136/266] 10.222.0.36 () {56 vars in 1120 bytes} [Mon Oct 31 14:43:50 2022] GET /identity/v3/roles/member => generated 84 bytes in 56 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:50.989961 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323d0697-ad97-46fd-9e9a-556dc917acab admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:51.006556 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323d0697-ad97-46fd-9e9a-556dc917acab admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:51.020745 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a986032-b0fb-497d-a21f-42a5d08876fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:51.022425 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a986032-b0fb-497d-a21f-42a5d08876fa admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:51.024487 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a986032-b0fb-497d-a21f-42a5d08876fa admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:51.024965 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a986032-b0fb-497d-a21f-42a5d08876fa admin admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c/groups/496de8c8aebb422fbcae2f9e32bba846/roles/addf152a48404c27b7d71386e81b8f76` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:51.026339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4d2c48-e6f0-4dce-84ec-a1e92e7ff126 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:51.027660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a986032-b0fb-497d-a21f-42a5d08876fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc35ec7707ee4a09b3784f8b3e76686c, group_id=496de8c8aebb422fbcae2f9e32bba846, role_id=addf152a48404c27b7d71386e81b8f76)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:51.028900 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: WARNING py.warnings [None req-6a986032-b0fb-497d-a21f-42a5d08876fa admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 31 14:43:51.028900 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: warnings.warn(msg) Oct 31 14:43:51.028900 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: Oct 31 14:43:51.031220 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a986032-b0fb-497d-a21f-42a5d08876fa admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:51.050848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 137/267] 10.222.0.36 () {58 vars in 1463 bytes} [Mon Oct 31 14:43:50 2022] PUT /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c/groups/496de8c8aebb422fbcae2f9e32bba846/roles/addf152a48404c27b7d71386e81b8f76 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:43:51.053571 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a986032-b0fb-497d-a21f-42a5d08876fa admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:51.073860 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd78bc29-d877-41a7-b0dc-7b84401416d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:51.074497 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd78bc29-d877-41a7-b0dc-7b84401416d3 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:51.074769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd78bc29-d877-41a7-b0dc-7b84401416d3 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:51.075117 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd78bc29-d877-41a7-b0dc-7b84401416d3 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:51.087516 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd78bc29-d877-41a7-b0dc-7b84401416d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:51.093205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-fd78bc29-d877-41a7-b0dc-7b84401416d3 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 31 14:43:51.093205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:51.093205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:51.093205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:51.093205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:51.093205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:51.093205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:51.093205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:51.093205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:51.093205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:51.093205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:51.093205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:51.093205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:51.093205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:51.093205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:51.093205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:51.094128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:51.094128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:51.094128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:51.094128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:51.094128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:51.094128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:51.094128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:51.094128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:51.094128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:51.094128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:51.094128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:51.094128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:51.094128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:51.094128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:51.094128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:51.094128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:51.094903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:51.094903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:51.094903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:51.094903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:51.094903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:51.094903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:51.094903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:51.094903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:51.094903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:51.094903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:51.094903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 31 14:43:51.094903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:51.094903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 131/268] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:50 2022] GET /identity/v3/roles/reader => generated 84 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:43:51.097385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e0271f-3e56-4764-bddc-fa704295e879 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:51.097385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e0271f-3e56-4764-bddc-fa704295e879 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:51.097385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e0271f-3e56-4764-bddc-fa704295e879 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:51.099633 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:51.100785 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 132/269] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:51 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:51.106978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e0271f-3e56-4764-bddc-fa704295e879 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:51.150231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a986032-b0fb-497d-a21f-42a5d08876fa admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:51.154469 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f4b93d-352d-421d-8042-604de120b25e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:51.155756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f4b93d-352d-421d-8042-604de120b25e admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:51.156037 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f4b93d-352d-421d-8042-604de120b25e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:51.156369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f4b93d-352d-421d-8042-604de120b25e admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:51.157852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f4b93d-352d-421d-8042-604de120b25e admin admin] RBAC: Adding query filter params (name=member) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:51.157852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f4b93d-352d-421d-8042-604de120b25e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:51.159614 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f4b93d-352d-421d-8042-604de120b25e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:51.172632 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 138/270] 10.222.0.36 () {56 vars in 1122 bytes} [Mon Oct 31 14:43:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:51.174377 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19450fae-c625-4cc5-b62b-d6fea8a8a66f None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:51.174665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19450fae-c625-4cc5-b62b-d6fea8a8a66f None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:51.175032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19450fae-c625-4cc5-b62b-d6fea8a8a66f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:51.199577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-19450fae-c625-4cc5-b62b-d6fea8a8a66f None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:51.202175 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-19450fae-c625-4cc5-b62b-d6fea8a8a66f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:51.211103 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 139/271] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:51 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 31 14:43:51.212886 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19450fae-c625-4cc5-b62b-d6fea8a8a66f None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:51.218839 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:51.225988 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19450fae-c625-4cc5-b62b-d6fea8a8a66f None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:51.232553 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fe9036-0c38-41a7-af5d-d096d6714494 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:51.234547 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fe9036-0c38-41a7-af5d-d096d6714494 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:51.235073 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fe9036-0c38-41a7-af5d-d096d6714494 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:51.235585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fe9036-0c38-41a7-af5d-d096d6714494 admin admin] PATH_INFO: `/v3/groups/496de8c8aebb422fbcae2f9e32bba846` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:51.237191 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fe9036-0c38-41a7-af5d-d096d6714494 admin admin] RBAC: Authorizing `identity:get_group(group_id=496de8c8aebb422fbcae2f9e32bba846)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:51.239741 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fe9036-0c38-41a7-af5d-d096d6714494 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:51.241875 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 140/272] 10.222.0.36 () {56 vars in 1201 bytes} [Mon Oct 31 14:43:51 2022] GET /identity/v3/groups/496de8c8aebb422fbcae2f9e32bba846 => generated 226 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:51.247625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fe9036-0c38-41a7-af5d-d096d6714494 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:51.267009 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fe9036-0c38-41a7-af5d-d096d6714494 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:51.273052 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e0271f-3e56-4764-bddc-fa704295e879 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:51.282722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd49f31-8afd-4f59-b405-50f62c5a3893 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:51.284113 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd49f31-8afd-4f59-b405-50f62c5a3893 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:51.285408 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd49f31-8afd-4f59-b405-50f62c5a3893 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:51.285891 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd49f31-8afd-4f59-b405-50f62c5a3893 admin admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:51.288331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd49f31-8afd-4f59-b405-50f62c5a3893 admin admin] RBAC: Authorizing `identity:get_project(project_id=dc35ec7707ee4a09b3784f8b3e76686c)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:51.293212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd49f31-8afd-4f59-b405-50f62c5a3893 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:51.296415 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 141/273] 10.222.0.36 () {56 vars in 1207 bytes} [Mon Oct 31 14:43:51 2022] GET /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c => generated 298 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:51.302929 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd49f31-8afd-4f59-b405-50f62c5a3893 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:51.320430 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd49f31-8afd-4f59-b405-50f62c5a3893 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:51.332131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f759e722-2ae7-486f-aa71-a337aaf250fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:51.333059 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f759e722-2ae7-486f-aa71-a337aaf250fd admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:51.334699 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f759e722-2ae7-486f-aa71-a337aaf250fd admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:51.335459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f759e722-2ae7-486f-aa71-a337aaf250fd admin admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c/groups/496de8c8aebb422fbcae2f9e32bba846/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:51.339435 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f759e722-2ae7-486f-aa71-a337aaf250fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc35ec7707ee4a09b3784f8b3e76686c, group_id=496de8c8aebb422fbcae2f9e32bba846, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:51.343829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f759e722-2ae7-486f-aa71-a337aaf250fd admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:51.348390 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d31cd2-5af1-453b-a09d-a26435f4d9e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:51.350881 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d31cd2-5af1-453b-a09d-a26435f4d9e1 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:51.351426 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d31cd2-5af1-453b-a09d-a26435f4d9e1 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:51.351915 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d31cd2-5af1-453b-a09d-a26435f4d9e1 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:51.354135 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d31cd2-5af1-453b-a09d-a26435f4d9e1 None admin] RBAC: Adding query filter params (name=reader) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:51.354652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d31cd2-5af1-453b-a09d-a26435f4d9e1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:51.356889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d31cd2-5af1-453b-a09d-a26435f4d9e1 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:51.371444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 133/274] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:51 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:51.377780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d31cd2-5af1-453b-a09d-a26435f4d9e1 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:51.380712 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 142/275] 10.222.0.36 () {58 vars in 1463 bytes} [Mon Oct 31 14:43:51 2022] PUT /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c/groups/496de8c8aebb422fbcae2f9e32bba846/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:43:51.500547 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d31cd2-5af1-453b-a09d-a26435f4d9e1 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:51.551187 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b1289c-d64f-44b0-a553-96f31cc9bd6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:51.552273 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b1289c-d64f-44b0-a553-96f31cc9bd6b None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:51.552768 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b1289c-d64f-44b0-a553-96f31cc9bd6b None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:51.553285 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b1289c-d64f-44b0-a553-96f31cc9bd6b None admin] PATH_INFO: `/v3/users/d0ebd64a902d42f3bf9133cccfc71c2e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:51.554704 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b1289c-d64f-44b0-a553-96f31cc9bd6b None admin] RBAC: Authorizing `identity:get_user(user_id=d0ebd64a902d42f3bf9133cccfc71c2e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:51.556478 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b1289c-d64f-44b0-a553-96f31cc9bd6b None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:51.558314 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 134/276] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:43:51 2022] GET /identity/v3/users/d0ebd64a902d42f3bf9133cccfc71c2e => generated 293 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:51.568575 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f759e722-2ae7-486f-aa71-a337aaf250fd admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:51.670969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b1289c-d64f-44b0-a553-96f31cc9bd6b None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:51.678232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:51.704938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f759e722-2ae7-486f-aa71-a337aaf250fd admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:51.752283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4715aa11-69e3-4c58-89c9-002e8d4d906b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:51.752893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4715aa11-69e3-4c58-89c9-002e8d4d906b None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:51.753171 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4715aa11-69e3-4c58-89c9-002e8d4d906b None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:51.753439 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4715aa11-69e3-4c58-89c9-002e8d4d906b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:51.754171 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4715aa11-69e3-4c58-89c9-002e8d4d906b None admin] RBAC: Adding query filter params (role.id=d5174defb0fa4869819ae433429cb2df, user.id=d0ebd64a902d42f3bf9133cccfc71c2e, scope.system=all) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:51.754475 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4715aa11-69e3-4c58-89c9-002e8d4d906b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:51.756224 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4715aa11-69e3-4c58-89c9-002e8d4d906b None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:51.767839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:51.772453 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 143/277] 10.222.0.36 () {56 vars in 1308 bytes} [Mon Oct 31 14:43:51 2022] GET /identity/v3/role_assignments?user.id=d0ebd64a902d42f3bf9133cccfc71c2e&scope.system=all&role.id=d5174defb0fa4869819ae433429cb2df => generated 510 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:51.786618 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b1289c-d64f-44b0-a553-96f31cc9bd6b None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:51.795286 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efcda608-f379-4581-ac7c-82813278e82a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:51.796386 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcda608-f379-4581-ac7c-82813278e82a admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:51.796873 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcda608-f379-4581-ac7c-82813278e82a admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:51.797339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcda608-f379-4581-ac7c-82813278e82a admin admin] PATH_INFO: `/v3/roles/member` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:51.824712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efcda608-f379-4581-ac7c-82813278e82a admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:51.830755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-efcda608-f379-4581-ac7c-82813278e82a admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:51.830755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:51.830755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:51.830755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:51.830755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:51.830755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:51.830755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:51.830755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:51.830755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:51.830755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:51.830755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:51.830755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:51.830755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:51.830755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:51.830755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:51.830755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:51.832133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:51.832133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:51.832133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:51.832133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:51.832133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:51.832133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:51.832133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:51.832133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:51.832133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:51.832133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:51.832133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:51.832133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:51.832133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:51.832133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:51.832133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:51.832133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:51.832926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:51.832926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:51.832926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:51.832926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:51.832926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:51.832926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:51.832926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:51.832926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:51.832926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:51.832926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:51.832926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:51.832926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:51.835350 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 135/278] 10.222.0.36 () {56 vars in 1120 bytes} [Mon Oct 31 14:43:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:43:51.841294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4715aa11-69e3-4c58-89c9-002e8d4d906b None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:51.864553 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4715aa11-69e3-4c58-89c9-002e8d4d906b None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:51.870798 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8b63b5-312a-42bb-b300-dc23b7bac620 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:51.871464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8b63b5-312a-42bb-b300-dc23b7bac620 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:51.871768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8b63b5-312a-42bb-b300-dc23b7bac620 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:51.872042 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8b63b5-312a-42bb-b300-dc23b7bac620 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:51.872768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8b63b5-312a-42bb-b300-dc23b7bac620 admin admin] RBAC: Adding query filter params (name=member) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:51.873069 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8b63b5-312a-42bb-b300-dc23b7bac620 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:51.875645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8b63b5-312a-42bb-b300-dc23b7bac620 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:51.895191 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 144/279] 10.222.0.36 () {56 vars in 1122 bytes} [Mon Oct 31 14:43:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:51.901360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efcda608-f379-4581-ac7c-82813278e82a admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:51.916850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efcda608-f379-4581-ac7c-82813278e82a admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:51.927237 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d2473e-b875-4604-86e4-9e80bb7aa561 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:51.928287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d2473e-b875-4604-86e4-9e80bb7aa561 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:51.928762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d2473e-b875-4604-86e4-9e80bb7aa561 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:51.929228 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d2473e-b875-4604-86e4-9e80bb7aa561 admin admin] PATH_INFO: `/v3/groups/496de8c8aebb422fbcae2f9e32bba846` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:51.930630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d2473e-b875-4604-86e4-9e80bb7aa561 admin admin] RBAC: Authorizing `identity:get_group(group_id=496de8c8aebb422fbcae2f9e32bba846)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:51.932713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: WARNING py.warnings [None req-e3d2473e-b875-4604-86e4-9e80bb7aa561 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 31 14:43:51.932713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: warnings.warn(msg) Oct 31 14:43:51.932713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: Oct 31 14:43:51.934301 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d2473e-b875-4604-86e4-9e80bb7aa561 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:51.936817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 136/280] 10.222.0.36 () {56 vars in 1201 bytes} [Mon Oct 31 14:43:51 2022] GET /identity/v3/groups/496de8c8aebb422fbcae2f9e32bba846 => generated 226 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:51.942462 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8b63b5-312a-42bb-b300-dc23b7bac620 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:51.956137 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8b63b5-312a-42bb-b300-dc23b7bac620 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:51.962685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5ad740-ed64-415b-9e06-6e96112ae96a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:51.963419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5ad740-ed64-415b-9e06-6e96112ae96a admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:51.963732 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5ad740-ed64-415b-9e06-6e96112ae96a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:51.964007 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5ad740-ed64-415b-9e06-6e96112ae96a admin admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:51.966281 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5ad740-ed64-415b-9e06-6e96112ae96a admin admin] RBAC: Authorizing `identity:get_project(project_id=5847498d7fa84446a3389fbb99498368)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:51.968600 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5ad740-ed64-415b-9e06-6e96112ae96a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:51.970914 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 145/281] 10.222.0.36 () {56 vars in 1207 bytes} [Mon Oct 31 14:43:51 2022] GET /identity/v3/projects/5847498d7fa84446a3389fbb99498368 => generated 302 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:51.976732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d2473e-b875-4604-86e4-9e80bb7aa561 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:51.993123 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d2473e-b875-4604-86e4-9e80bb7aa561 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:52.003473 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17ce200-60f3-4090-ba5c-33afacf3a1f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:52.004524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17ce200-60f3-4090-ba5c-33afacf3a1f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:52.004998 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17ce200-60f3-4090-ba5c-33afacf3a1f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:52.005482 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17ce200-60f3-4090-ba5c-33afacf3a1f7 admin admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368/groups/496de8c8aebb422fbcae2f9e32bba846/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:52.007706 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17ce200-60f3-4090-ba5c-33afacf3a1f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5847498d7fa84446a3389fbb99498368, group_id=496de8c8aebb422fbcae2f9e32bba846, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:52.008962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: WARNING py.warnings [None req-d17ce200-60f3-4090-ba5c-33afacf3a1f7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 31 14:43:52.008962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: warnings.warn(msg) Oct 31 14:43:52.008962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: Oct 31 14:43:52.011259 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17ce200-60f3-4090-ba5c-33afacf3a1f7 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:52.032705 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 137/282] 10.222.0.36 () {58 vars in 1463 bytes} [Mon Oct 31 14:43:51 2022] PUT /identity/v3/projects/5847498d7fa84446a3389fbb99498368/groups/496de8c8aebb422fbcae2f9e32bba846/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:43:52.712803 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0e3857-a673-411f-b1c1-8c4bdce40cca None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:52.713308 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0e3857-a673-411f-b1c1-8c4bdce40cca None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:52.713606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0e3857-a673-411f-b1c1-8c4bdce40cca None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:52.714786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 146/283] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:52.728683 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32957af3-a27a-4e9c-8ff0-425fb949dfa7 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:52.729149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32957af3-a27a-4e9c-8ff0-425fb949dfa7 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:52.729229 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32957af3-a27a-4e9c-8ff0-425fb949dfa7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:52.758373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-32957af3-a27a-4e9c-8ff0-425fb949dfa7 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:52.759482 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-32957af3-a27a-4e9c-8ff0-425fb949dfa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:52.840747 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 138/284] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:52 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 31 14:43:52.908049 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a593e35-efc1-4ccb-85f5-192984788585 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:52.908344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a593e35-efc1-4ccb-85f5-192984788585 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:52.908600 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a593e35-efc1-4ccb-85f5-192984788585 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:52.910004 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 147/285] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:52.925426 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6679c4-f4d8-4b77-91f4-4d415faec4b9 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:52.925712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6679c4-f4d8-4b77-91f4-4d415faec4b9 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:52.925989 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6679c4-f4d8-4b77-91f4-4d415faec4b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:52.948903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-8a6679c4-f4d8-4b77-91f4-4d415faec4b9 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:52.950100 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-8a6679c4-f4d8-4b77-91f4-4d415faec4b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:52.958569 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 139/286] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:52 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 31 14:43:53.373119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a593e35-efc1-4ccb-85f5-192984788585 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:53.387473 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a593e35-efc1-4ccb-85f5-192984788585 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:53.395805 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd6b6e2-7045-4e53-b236-da635777526a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:53.396431 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd6b6e2-7045-4e53-b236-da635777526a admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:53.396711 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd6b6e2-7045-4e53-b236-da635777526a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:53.396983 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd6b6e2-7045-4e53-b236-da635777526a admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:53.413965 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd6b6e2-7045-4e53-b236-da635777526a admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:53.419297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-dbd6b6e2-7045-4e53-b236-da635777526a admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:43:53.419297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:53.419297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:53.419297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:53.419297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:53.419297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:53.419297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:53.419297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:53.419297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:53.419297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:53.419297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:53.419297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:53.419297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:53.419297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:53.419297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:53.419297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:53.421634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:53.421634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:53.421634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:53.421634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:53.421634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:53.421634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:53.421634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:53.421634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:53.421634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:53.421634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:53.421634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:53.421634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:53.421634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:53.421634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:53.421634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:53.421634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:53.423520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:53.423520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:53.423520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:53.423520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:53.423520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:53.423520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:53.423520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:53.423520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:53.423520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:53.423520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:53.423520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:43:53.423520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:53.423520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 148/287] 10.222.0.36 () {56 vars in 1135 bytes} [Mon Oct 31 14:43:53 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 48 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:53.428809 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6679c4-f4d8-4b77-91f4-4d415faec4b9 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:53.442615 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6679c4-f4d8-4b77-91f4-4d415faec4b9 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:53.447296 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc5ba36-8102-4c2c-b522-e92144306c34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:53.447932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc5ba36-8102-4c2c-b522-e92144306c34 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:53.448211 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc5ba36-8102-4c2c-b522-e92144306c34 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:53.448484 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc5ba36-8102-4c2c-b522-e92144306c34 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:53.449160 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc5ba36-8102-4c2c-b522-e92144306c34 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:53.449461 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc5ba36-8102-4c2c-b522-e92144306c34 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:53.450538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: WARNING py.warnings [None req-dcc5ba36-8102-4c2c-b522-e92144306c34 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 31 14:43:53.450538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: warnings.warn(msg) Oct 31 14:43:53.450538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: Oct 31 14:43:53.451488 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc5ba36-8102-4c2c-b522-e92144306c34 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:53.460917 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 140/288] 10.222.0.36 () {56 vars in 1132 bytes} [Mon Oct 31 14:43:53 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:53.466380 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd6b6e2-7045-4e53-b236-da635777526a admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:53.473275 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc5ba36-8102-4c2c-b522-e92144306c34 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:53.484133 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd6b6e2-7045-4e53-b236-da635777526a admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:53.491410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3679b5a0-c781-47d7-97e8-045437663c5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:53.492097 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3679b5a0-c781-47d7-97e8-045437663c5b admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:53.492378 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3679b5a0-c781-47d7-97e8-045437663c5b admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:53.492650 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3679b5a0-c781-47d7-97e8-045437663c5b admin admin] PATH_INFO: `/v3/roles/admin` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:53.500060 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc5ba36-8102-4c2c-b522-e92144306c34 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:53.510891 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d6eb35-ae6d-4e9f-9fa6-01d1bb66448c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:53.512466 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d6eb35-ae6d-4e9f-9fa6-01d1bb66448c admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:53.512760 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d6eb35-ae6d-4e9f-9fa6-01d1bb66448c admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:53.513034 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d6eb35-ae6d-4e9f-9fa6-01d1bb66448c admin admin] PATH_INFO: `/v3/groups/496de8c8aebb422fbcae2f9e32bba846` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:53.516342 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d6eb35-ae6d-4e9f-9fa6-01d1bb66448c admin admin] RBAC: Authorizing `identity:get_group(group_id=496de8c8aebb422fbcae2f9e32bba846)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:53.525236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d6eb35-ae6d-4e9f-9fa6-01d1bb66448c admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:53.527584 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3679b5a0-c781-47d7-97e8-045437663c5b admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:53.538355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-3679b5a0-c781-47d7-97e8-045437663c5b admin admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:43:53.538355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:53.538355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:53.538355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:53.538355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:53.538355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:53.538355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:53.538355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:53.538355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:53.538355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:53.538355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:53.538355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:53.538355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:53.538355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:53.538355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:53.538355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:53.540512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:53.540512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:53.540512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:53.540512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:53.540512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:53.540512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:53.540512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:53.540512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:53.540512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:53.540512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:53.540512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:53.540512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:53.540512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:53.540512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:53.540512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:53.540512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:53.542640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:53.542640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:53.542640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:53.542640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:53.542640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:53.542640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:53.542640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:53.542640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:53.542640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:53.542640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:53.542640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:43:53.542640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:53.544139 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 141/289] 10.222.0.36 () {56 vars in 1201 bytes} [Mon Oct 31 14:43:53 2022] GET /identity/v3/groups/496de8c8aebb422fbcae2f9e32bba846 => generated 226 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:53.544139 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d6eb35-ae6d-4e9f-9fa6-01d1bb66448c admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:53.544426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 149/290] 10.222.0.36 () {56 vars in 1117 bytes} [Mon Oct 31 14:43:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 75 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:53.554745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d6eb35-ae6d-4e9f-9fa6-01d1bb66448c admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:53.558029 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3679b5a0-c781-47d7-97e8-045437663c5b admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:53.571671 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3679b5a0-c781-47d7-97e8-045437663c5b admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:53.574573 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4420f3fc-9736-4943-9e5a-76847755e105 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:53.575263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4420f3fc-9736-4943-9e5a-76847755e105 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:53.575570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4420f3fc-9736-4943-9e5a-76847755e105 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:53.575878 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4420f3fc-9736-4943-9e5a-76847755e105 admin admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:53.577845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4420f3fc-9736-4943-9e5a-76847755e105 admin admin] RBAC: Authorizing `identity:get_project(project_id=5847498d7fa84446a3389fbb99498368)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:53.580937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4420f3fc-9736-4943-9e5a-76847755e105 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:53.584200 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6636a79d-6e41-4522-a23d-4179ad93b87a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:53.584864 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6636a79d-6e41-4522-a23d-4179ad93b87a admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:53.585146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6636a79d-6e41-4522-a23d-4179ad93b87a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:53.585528 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6636a79d-6e41-4522-a23d-4179ad93b87a admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:53.586255 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6636a79d-6e41-4522-a23d-4179ad93b87a admin admin] RBAC: Adding query filter params (name=admin) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:53.586552 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6636a79d-6e41-4522-a23d-4179ad93b87a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:53.587191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 142/291] 10.222.0.36 () {56 vars in 1207 bytes} [Mon Oct 31 14:43:53 2022] GET /identity/v3/projects/5847498d7fa84446a3389fbb99498368 => generated 302 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:53.589491 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4420f3fc-9736-4943-9e5a-76847755e105 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:53.589857 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6636a79d-6e41-4522-a23d-4179ad93b87a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:53.608384 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 150/292] 10.222.0.36 () {56 vars in 1120 bytes} [Mon Oct 31 14:43:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:53.613797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4420f3fc-9736-4943-9e5a-76847755e105 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:53.618464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc970aa8-e253-4f52-ba2d-b4f068cde94d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:53.619124 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc970aa8-e253-4f52-ba2d-b4f068cde94d admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:53.619403 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc970aa8-e253-4f52-ba2d-b4f068cde94d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:53.619706 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc970aa8-e253-4f52-ba2d-b4f068cde94d admin admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368/groups/496de8c8aebb422fbcae2f9e32bba846/roles/addf152a48404c27b7d71386e81b8f76` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:53.622127 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6636a79d-6e41-4522-a23d-4179ad93b87a admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:53.630404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc970aa8-e253-4f52-ba2d-b4f068cde94d admin admin] RBAC: Authorizing `identity:create_grant(project_id=5847498d7fa84446a3389fbb99498368, group_id=496de8c8aebb422fbcae2f9e32bba846, role_id=addf152a48404c27b7d71386e81b8f76)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:53.634129 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc970aa8-e253-4f52-ba2d-b4f068cde94d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:53.638080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6636a79d-6e41-4522-a23d-4179ad93b87a admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:53.646717 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b8c757-93b5-43b7-b03c-53b668763e46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:53.647402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b8c757-93b5-43b7-b03c-53b668763e46 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:53.647792 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b8c757-93b5-43b7-b03c-53b668763e46 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:53.648132 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b8c757-93b5-43b7-b03c-53b668763e46 admin admin] PATH_INFO: `/v3/groups/28b8df505ef941b0a9b8f50e79f5986c` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:53.659625 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 143/293] 10.222.0.36 () {58 vars in 1463 bytes} [Mon Oct 31 14:43:53 2022] PUT /identity/v3/projects/5847498d7fa84446a3389fbb99498368/groups/496de8c8aebb422fbcae2f9e32bba846/roles/addf152a48404c27b7d71386e81b8f76 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:43:53.662752 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b8c757-93b5-43b7-b03c-53b668763e46 admin admin] RBAC: Authorizing `identity:get_group(group_id=28b8df505ef941b0a9b8f50e79f5986c)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:53.664786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b8c757-93b5-43b7-b03c-53b668763e46 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:53.666609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 151/294] 10.222.0.36 () {56 vars in 1201 bytes} [Mon Oct 31 14:43:53 2022] GET /identity/v3/groups/28b8df505ef941b0a9b8f50e79f5986c => generated 229 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:53.672451 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc970aa8-e253-4f52-ba2d-b4f068cde94d admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:53.737227 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:53.760606 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc970aa8-e253-4f52-ba2d-b4f068cde94d admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:53.770510 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeab5e3d-4715-4467-84e2-fd895234c094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:53.771218 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeab5e3d-4715-4467-84e2-fd895234c094 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:53.771503 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeab5e3d-4715-4467-84e2-fd895234c094 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:53.771803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeab5e3d-4715-4467-84e2-fd895234c094 admin admin] PATH_INFO: `/v3/projects/e18b2b57837347e48db3adc0214764b8` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:53.773010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeab5e3d-4715-4467-84e2-fd895234c094 admin admin] RBAC: Authorizing `identity:get_project(project_id=e18b2b57837347e48db3adc0214764b8)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:53.774863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeab5e3d-4715-4467-84e2-fd895234c094 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:53.777008 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 144/295] 10.222.0.36 () {56 vars in 1207 bytes} [Mon Oct 31 14:43:53 2022] GET /identity/v3/projects/e18b2b57837347e48db3adc0214764b8 => generated 344 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:53.784120 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b8c757-93b5-43b7-b03c-53b668763e46 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:53.794927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b8c757-93b5-43b7-b03c-53b668763e46 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:53.803984 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb299506-cfe4-493e-926a-3e8ee1d9cc31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:53.804672 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb299506-cfe4-493e-926a-3e8ee1d9cc31 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:53.804953 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb299506-cfe4-493e-926a-3e8ee1d9cc31 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:53.805228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb299506-cfe4-493e-926a-3e8ee1d9cc31 admin admin] PATH_INFO: `/v3/projects/e18b2b57837347e48db3adc0214764b8/groups/28b8df505ef941b0a9b8f50e79f5986c/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:53.810850 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb299506-cfe4-493e-926a-3e8ee1d9cc31 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e18b2b57837347e48db3adc0214764b8, group_id=28b8df505ef941b0a9b8f50e79f5986c, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:53.813670 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb299506-cfe4-493e-926a-3e8ee1d9cc31 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:53.834330 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 152/296] 10.222.0.36 () {58 vars in 1463 bytes} [Mon Oct 31 14:43:53 2022] PUT /identity/v3/projects/e18b2b57837347e48db3adc0214764b8/groups/28b8df505ef941b0a9b8f50e79f5986c/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:43:54.182867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33f0d59-5ad3-4008-a496-b161e6a11330 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:54.183185 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33f0d59-5ad3-4008-a496-b161e6a11330 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:54.183510 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33f0d59-5ad3-4008-a496-b161e6a11330 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:54.185039 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 145/297] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:54.191885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cdb218-ba39-4c4f-a25c-e97cd62582a9 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:54.192163 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cdb218-ba39-4c4f-a25c-e97cd62582a9 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:54.192440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cdb218-ba39-4c4f-a25c-e97cd62582a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:54.217545 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-53cdb218-ba39-4c4f-a25c-e97cd62582a9 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:54.219756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-53cdb218-ba39-4c4f-a25c-e97cd62582a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:54.244993 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9b529a-d118-4342-a45d-621c47ea95e9 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:54.245295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9b529a-d118-4342-a45d-621c47ea95e9 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:54.245538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9b529a-d118-4342-a45d-621c47ea95e9 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:54.247191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 146/298] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:54.254190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1088eeaf-2977-408b-a498-f3f1fdd443a3 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:54.254468 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1088eeaf-2977-408b-a498-f3f1fdd443a3 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:54.254774 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1088eeaf-2977-408b-a498-f3f1fdd443a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:54.294827 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-1088eeaf-2977-408b-a498-f3f1fdd443a3 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:54.296149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-1088eeaf-2977-408b-a498-f3f1fdd443a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:54.314025 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 153/299] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:54.422862 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 147/300] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:43:54.632105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cdb218-ba39-4c4f-a25c-e97cd62582a9 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:54.706135 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:54.759649 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cdb218-ba39-4c4f-a25c-e97cd62582a9 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:54.797285 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5836ad-35fc-46e3-8aaf-a74ba03e0181 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:54.797898 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5836ad-35fc-46e3-8aaf-a74ba03e0181 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:54.798137 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5836ad-35fc-46e3-8aaf-a74ba03e0181 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:54.798355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5836ad-35fc-46e3-8aaf-a74ba03e0181 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:54.811455 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5836ad-35fc-46e3-8aaf-a74ba03e0181 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:54.816764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-ae5836ad-35fc-46e3-8aaf-a74ba03e0181 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:43:54.816764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:54.816764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:54.816764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:54.816764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:54.816764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:54.816764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:54.816764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:54.816764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:54.816764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:54.816764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:54.816764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:54.816764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:54.816764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:54.816764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:54.816764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:54.817645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:54.817645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:54.817645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:54.817645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:54.817645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:54.817645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:54.817645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:54.817645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:54.817645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:54.817645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:54.817645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:54.817645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:54.817645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:54.817645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:54.817645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:54.817645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:54.818426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:54.818426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:54.818426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:54.818426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:54.818426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:54.818426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:54.818426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:54.818426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:54.818426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:54.818426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:54.818426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:43:54.818426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:54.819019 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1088eeaf-2977-408b-a498-f3f1fdd443a3 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:54.819403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 154/301] 10.222.0.36 () {56 vars in 1114 bytes} [Mon Oct 31 14:43:54 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:54.825881 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5836ad-35fc-46e3-8aaf-a74ba03e0181 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:54.986046 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5836ad-35fc-46e3-8aaf-a74ba03e0181 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:55.018543 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:55.036032 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1088eeaf-2977-408b-a498-f3f1fdd443a3 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:55.040973 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df8787c-d203-4631-b5f7-496877171d28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:55.041583 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df8787c-d203-4631-b5f7-496877171d28 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:55.041860 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df8787c-d203-4631-b5f7-496877171d28 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:55.042129 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df8787c-d203-4631-b5f7-496877171d28 None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:55.042792 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df8787c-d203-4631-b5f7-496877171d28 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:55.043107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df8787c-d203-4631-b5f7-496877171d28 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:55.044673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df8787c-d203-4631-b5f7-496877171d28 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:55.054327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 155/302] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:43:54 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:55.055527 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:55.056712 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa48df6a-0cfa-42b0-8c47-8b9bfc6c8c1f None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:55.057265 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa48df6a-0cfa-42b0-8c47-8b9bfc6c8c1f None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:55.057679 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa48df6a-0cfa-42b0-8c47-8b9bfc6c8c1f None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:55.061834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 156/303] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:55 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:55.063517 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa48df6a-0cfa-42b0-8c47-8b9bfc6c8c1f None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:55.085298 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926cd5f6-c764-43c1-a90f-26822136698d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:55.085937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926cd5f6-c764-43c1-a90f-26822136698d None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:55.086214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926cd5f6-c764-43c1-a90f-26822136698d None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:55.086482 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926cd5f6-c764-43c1-a90f-26822136698d None admin] PATH_INFO: `/v3/roles/member` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:55.099191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926cd5f6-c764-43c1-a90f-26822136698d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:55.106047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-926cd5f6-c764-43c1-a90f-26822136698d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:55.106047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:55.106047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:55.106047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:55.106047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:55.106047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:55.106047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:55.106047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:55.106047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:55.106047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:55.106047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:55.106047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:55.106047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:55.106047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:55.106047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:55.106047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:55.107173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:55.107173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:55.107173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:55.107173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:55.107173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:55.107173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:55.107173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:55.107173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:55.107173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:55.107173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:55.107173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:55.107173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:55.107173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:55.107173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:55.107173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:55.107173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:55.108696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:55.108696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:55.108696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:55.108696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:55.108696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:55.108696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:55.108696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:55.108696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:55.108696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:55.108696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:55.108696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:55.108696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:55.108696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 148/304] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:55.108696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c435a4-3618-4371-b13d-0b2e5b5c86ad None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:55.109549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c435a4-3618-4371-b13d-0b2e5b5c86ad None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:55.109549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c435a4-3618-4371-b13d-0b2e5b5c86ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:55.129580 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-86c435a4-3618-4371-b13d-0b2e5b5c86ad None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:55.130786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-86c435a4-3618-4371-b13d-0b2e5b5c86ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:55.214554 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa48df6a-0cfa-42b0-8c47-8b9bfc6c8c1f None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:55.223862 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 149/305] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:55.225002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c435a4-3618-4371-b13d-0b2e5b5c86ad None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:55.260766 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5d4b52-ff9d-4919-9de3-9ed0fb0d27dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:55.261430 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5d4b52-ff9d-4919-9de3-9ed0fb0d27dd None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:55.261704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5d4b52-ff9d-4919-9de3-9ed0fb0d27dd None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:55.261971 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5d4b52-ff9d-4919-9de3-9ed0fb0d27dd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:55.335397 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:55.368266 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5d4b52-ff9d-4919-9de3-9ed0fb0d27dd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:55.370721 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5d4b52-ff9d-4919-9de3-9ed0fb0d27dd None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:55.379647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c435a4-3618-4371-b13d-0b2e5b5c86ad None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:55.393801 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:55.441854 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d876d5d8-4f29-4ce9-a298-731d2cd7a381 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:55.442474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d876d5d8-4f29-4ce9-a298-731d2cd7a381 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:55.442757 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d876d5d8-4f29-4ce9-a298-731d2cd7a381 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:55.443030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d876d5d8-4f29-4ce9-a298-731d2cd7a381 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:55.443732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d876d5d8-4f29-4ce9-a298-731d2cd7a381 None admin] RBAC: Adding query filter params (name=member) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:55.444039 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d876d5d8-4f29-4ce9-a298-731d2cd7a381 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:55.445470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d876d5d8-4f29-4ce9-a298-731d2cd7a381 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:55.457783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 150/306] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:55.464704 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d876d5d8-4f29-4ce9-a298-731d2cd7a381 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:55.569286 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 157/307] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:43:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 506 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 31 14:43:55.575151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5d4b52-ff9d-4919-9de3-9ed0fb0d27dd None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:55.636733 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d876d5d8-4f29-4ce9-a298-731d2cd7a381 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:55.681212 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d661ed-c77c-4640-9c47-ca0cb4b12b9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:55.682197 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d661ed-c77c-4640-9c47-ca0cb4b12b9c None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:55.683941 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d661ed-c77c-4640-9c47-ca0cb4b12b9c None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:55.684218 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d661ed-c77c-4640-9c47-ca0cb4b12b9c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:55.695904 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:55.724190 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5d4b52-ff9d-4919-9de3-9ed0fb0d27dd None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:55.735752 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:55.765754 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd2a43b-ab7c-446c-8de1-d33e9a6ddcea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:55.766376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd2a43b-ab7c-446c-8de1-d33e9a6ddcea None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:55.766654 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd2a43b-ab7c-446c-8de1-d33e9a6ddcea None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:55.766927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd2a43b-ab7c-446c-8de1-d33e9a6ddcea None admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:55.768176 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd2a43b-ab7c-446c-8de1-d33e9a6ddcea None admin] RBAC: Authorizing `identity:get_project(project_id=dc35ec7707ee4a09b3784f8b3e76686c)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:55.769520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd2a43b-ab7c-446c-8de1-d33e9a6ddcea None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:55.771455 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 158/308] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:55 2022] GET /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c => generated 298 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:55.772246 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd2a43b-ab7c-446c-8de1-d33e9a6ddcea None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:55.777119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d661ed-c77c-4640-9c47-ca0cb4b12b9c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:55.778826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d661ed-c77c-4640-9c47-ca0cb4b12b9c None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:55.877360 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd2a43b-ab7c-446c-8de1-d33e9a6ddcea None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:55.891338 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 151/309] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:43:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 31 14:43:55.892932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d661ed-c77c-4640-9c47-ca0cb4b12b9c None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:55.918199 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ae13a2-9b84-42f6-8127-bd71b6a89063 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:55.918865 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ae13a2-9b84-42f6-8127-bd71b6a89063 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:55.919149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ae13a2-9b84-42f6-8127-bd71b6a89063 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:55.919420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ae13a2-9b84-42f6-8127-bd71b6a89063 None admin] PATH_INFO: `/v3/roles/member` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:55.930500 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ae13a2-9b84-42f6-8127-bd71b6a89063 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:55.935313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-33ae13a2-9b84-42f6-8127-bd71b6a89063 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:55.935313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:55.935313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:55.935313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:55.935313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:55.935313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:55.935313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:55.935313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:55.935313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:55.935313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:55.935313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:55.935313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:55.935313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:55.935313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:55.935313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:55.935313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:55.936746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:55.936746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:55.936746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:55.936746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:55.936746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:55.936746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:55.936746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:55.936746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:55.936746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:55.936746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:55.936746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:55.936746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:55.936746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:55.936746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:55.936746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:55.936746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:55.937597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:55.937597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:55.937597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:55.937597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:55.937597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:55.937597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:55.937597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:55.937597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:55.937597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:55.937597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:55.937597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 31 14:43:55.937597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:55.937597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 159/310] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:55.938263 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ae13a2-9b84-42f6-8127-bd71b6a89063 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:55.950612 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:55.975604 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:56.016406 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d661ed-c77c-4640-9c47-ca0cb4b12b9c None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:56.045000 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ae13a2-9b84-42f6-8127-bd71b6a89063 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:56.061031 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47423f87-d551-4789-819c-f547265a9f77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:56.062302 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47423f87-d551-4789-819c-f547265a9f77 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:56.063271 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47423f87-d551-4789-819c-f547265a9f77 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:56.063780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47423f87-d551-4789-819c-f547265a9f77 None admin] PATH_INFO: `/v3/groups/496de8c8aebb422fbcae2f9e32bba846` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:56.065356 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47423f87-d551-4789-819c-f547265a9f77 None admin] RBAC: Authorizing `identity:get_group(group_id=496de8c8aebb422fbcae2f9e32bba846)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:56.067653 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47423f87-d551-4789-819c-f547265a9f77 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:56.069524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 152/311] 10.222.0.36 () {56 vars in 1180 bytes} [Mon Oct 31 14:43:55 2022] GET /identity/v3/groups/496de8c8aebb422fbcae2f9e32bba846 => generated 226 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:56.071138 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47423f87-d551-4789-819c-f547265a9f77 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:56.086417 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9c15af-c5b0-4e0f-af0d-2ea04a335855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:56.087144 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9c15af-c5b0-4e0f-af0d-2ea04a335855 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:56.087486 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9c15af-c5b0-4e0f-af0d-2ea04a335855 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:56.087849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9c15af-c5b0-4e0f-af0d-2ea04a335855 None admin] PATH_INFO: `/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:56.089118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9c15af-c5b0-4e0f-af0d-2ea04a335855 None admin] RBAC: Authorizing `identity:get_project(project_id=dc35ec7707ee4a09b3784f8b3e76686c)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:56.090759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9c15af-c5b0-4e0f-af0d-2ea04a335855 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:56.092770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 160/312] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:55 2022] GET /identity/v3/projects/dc35ec7707ee4a09b3784f8b3e76686c => generated 298 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:56.093621 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9c15af-c5b0-4e0f-af0d-2ea04a335855 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:56.184672 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:56.212432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9c15af-c5b0-4e0f-af0d-2ea04a335855 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:56.216299 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47423f87-d551-4789-819c-f547265a9f77 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:56.257292 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d422a6c1-51cb-45a8-af12-785762329a73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:56.257933 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d422a6c1-51cb-45a8-af12-785762329a73 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:56.258214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d422a6c1-51cb-45a8-af12-785762329a73 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:56.258472 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d422a6c1-51cb-45a8-af12-785762329a73 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:56.259238 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d422a6c1-51cb-45a8-af12-785762329a73 None admin] RBAC: Adding query filter params (name=member) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:56.259572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d422a6c1-51cb-45a8-af12-785762329a73 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:56.260897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39eb42cc-ac42-4348-8734-af19bdd09398 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:56.261112 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d422a6c1-51cb-45a8-af12-785762329a73 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:56.261902 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eb42cc-ac42-4348-8734-af19bdd09398 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:56.262653 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eb42cc-ac42-4348-8734-af19bdd09398 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:56.263123 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eb42cc-ac42-4348-8734-af19bdd09398 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:56.264609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39eb42cc-ac42-4348-8734-af19bdd09398 None admin] RBAC: Adding query filter params (group.id=496de8c8aebb422fbcae2f9e32bba846, role.id=addf152a48404c27b7d71386e81b8f76, scope.project.id=dc35ec7707ee4a09b3784f8b3e76686c) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:56.265604 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:56.266709 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39eb42cc-ac42-4348-8734-af19bdd09398 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:56.268408 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39eb42cc-ac42-4348-8734-af19bdd09398 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:56.272918 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 153/313] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:43:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:56.273775 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d422a6c1-51cb-45a8-af12-785762329a73 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:56.276726 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 161/314] 10.222.0.36 () {56 vars in 1376 bytes} [Mon Oct 31 14:43:56 2022] GET /identity/v3/role_assignments?group.id=496de8c8aebb422fbcae2f9e32bba846&scope.project.id=dc35ec7707ee4a09b3784f8b3e76686c&role.id=addf152a48404c27b7d71386e81b8f76 => generated 611 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:56.287066 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39eb42cc-ac42-4348-8734-af19bdd09398 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:56.442646 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39eb42cc-ac42-4348-8734-af19bdd09398 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:56.493060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c9c641-fc40-49f3-9abd-78e852fec3d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:56.493703 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c9c641-fc40-49f3-9abd-78e852fec3d6 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:56.493980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c9c641-fc40-49f3-9abd-78e852fec3d6 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:56.494247 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c9c641-fc40-49f3-9abd-78e852fec3d6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:56.502098 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d422a6c1-51cb-45a8-af12-785762329a73 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:56.507602 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:56.557773 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b41a7e-3ac8-447e-8dc5-92a6a271ae8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:56.558422 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b41a7e-3ac8-447e-8dc5-92a6a271ae8d None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:56.558821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b41a7e-3ac8-447e-8dc5-92a6a271ae8d None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:56.559147 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b41a7e-3ac8-447e-8dc5-92a6a271ae8d None admin] PATH_INFO: `/v3/groups/496de8c8aebb422fbcae2f9e32bba846` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:56.560349 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b41a7e-3ac8-447e-8dc5-92a6a271ae8d None admin] RBAC: Authorizing `identity:get_group(group_id=496de8c8aebb422fbcae2f9e32bba846)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:56.562345 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b41a7e-3ac8-447e-8dc5-92a6a271ae8d None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:56.564065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 154/315] 10.222.0.36 () {56 vars in 1180 bytes} [Mon Oct 31 14:43:56 2022] GET /identity/v3/groups/496de8c8aebb422fbcae2f9e32bba846 => generated 226 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:56.573819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b41a7e-3ac8-447e-8dc5-92a6a271ae8d None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:56.626829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c9c641-fc40-49f3-9abd-78e852fec3d6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:56.628460 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c9c641-fc40-49f3-9abd-78e852fec3d6 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:56.675686 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:56.722291 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b41a7e-3ac8-447e-8dc5-92a6a271ae8d None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:56.768386 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b006b42-f69f-43aa-8245-09c4477acfc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:56.768593 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 162/316] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:43:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 484 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 31 14:43:56.769001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b006b42-f69f-43aa-8245-09c4477acfc4 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:56.769285 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b006b42-f69f-43aa-8245-09c4477acfc4 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:56.769560 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b006b42-f69f-43aa-8245-09c4477acfc4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:56.770251 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e614813-544a-40da-a608-c95d8cce13fb None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:56.770430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b006b42-f69f-43aa-8245-09c4477acfc4 None admin] RBAC: Adding query filter params (group.id=496de8c8aebb422fbcae2f9e32bba846, role.id=bb354a7b796d467d91ff4b184011796e, scope.project.id=dc35ec7707ee4a09b3784f8b3e76686c) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:56.770528 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e614813-544a-40da-a608-c95d8cce13fb None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:56.770749 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b006b42-f69f-43aa-8245-09c4477acfc4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:56.770857 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e614813-544a-40da-a608-c95d8cce13fb None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:56.772269 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b006b42-f69f-43aa-8245-09c4477acfc4 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:56.772422 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 163/317] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:56.777883 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4dc0e4-8b44-452c-8ce2-39b7a69a7080 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:56.778163 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4dc0e4-8b44-452c-8ce2-39b7a69a7080 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:56.778433 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4dc0e4-8b44-452c-8ce2-39b7a69a7080 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:56.781734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 155/318] 10.222.0.36 () {56 vars in 1376 bytes} [Mon Oct 31 14:43:56 2022] GET /identity/v3/role_assignments?group.id=496de8c8aebb422fbcae2f9e32bba846&scope.project.id=dc35ec7707ee4a09b3784f8b3e76686c&role.id=bb354a7b796d467d91ff4b184011796e => generated 611 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:56.782513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b006b42-f69f-43aa-8245-09c4477acfc4 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:56.800990 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-6a4dc0e4-8b44-452c-8ce2-39b7a69a7080 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:56.802216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-6a4dc0e4-8b44-452c-8ce2-39b7a69a7080 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:56.855820 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:56.911222 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 164/319] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Oct 31 14:43:56.912636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b68fc13-f8fd-4bc7-8fde-11cc9a89520c None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:56.912921 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b68fc13-f8fd-4bc7-8fde-11cc9a89520c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:56.913201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b68fc13-f8fd-4bc7-8fde-11cc9a89520c None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:56.918792 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 165/320] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:43:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:43:56.921667 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1132f51d-be0b-4f5a-b01a-37f35f5012e1 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:56.923455 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1132f51d-be0b-4f5a-b01a-37f35f5012e1 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:56.923771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1132f51d-be0b-4f5a-b01a-37f35f5012e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:56.944485 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-1132f51d-be0b-4f5a-b01a-37f35f5012e1 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:43:56.944681 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b006b42-f69f-43aa-8245-09c4477acfc4 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:56.945724 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1132f51d-be0b-4f5a-b01a-37f35f5012e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:43:56.994418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:57.001738 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba1653f-b3be-43bd-9f48-a80a7c0e5d5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:57.001738 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba1653f-b3be-43bd-9f48-a80a7c0e5d5b None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:57.002015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba1653f-b3be-43bd-9f48-a80a7c0e5d5b None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:57.002551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba1653f-b3be-43bd-9f48-a80a7c0e5d5b None admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:57.005411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba1653f-b3be-43bd-9f48-a80a7c0e5d5b None admin] RBAC: Authorizing `identity:get_project(project_id=5847498d7fa84446a3389fbb99498368)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:57.007275 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba1653f-b3be-43bd-9f48-a80a7c0e5d5b None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:57.008888 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 156/321] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:56 2022] GET /identity/v3/projects/5847498d7fa84446a3389fbb99498368 => generated 302 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:57.015195 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba1653f-b3be-43bd-9f48-a80a7c0e5d5b None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:57.040727 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 166/322] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:43:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:43:57.127507 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba1653f-b3be-43bd-9f48-a80a7c0e5d5b None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:57.161984 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d5cbc1-a300-404f-a81b-e877a63d3c1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:57.162610 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d5cbc1-a300-404f-a81b-e877a63d3c1e None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:57.162890 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d5cbc1-a300-404f-a81b-e877a63d3c1e None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:57.163166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d5cbc1-a300-404f-a81b-e877a63d3c1e None admin] PATH_INFO: `/v3/groups/496de8c8aebb422fbcae2f9e32bba846` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:57.164296 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d5cbc1-a300-404f-a81b-e877a63d3c1e None admin] RBAC: Authorizing `identity:get_group(group_id=496de8c8aebb422fbcae2f9e32bba846)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:57.165823 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d5cbc1-a300-404f-a81b-e877a63d3c1e None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:57.167585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 157/323] 10.222.0.36 () {56 vars in 1180 bytes} [Mon Oct 31 14:43:57 2022] GET /identity/v3/groups/496de8c8aebb422fbcae2f9e32bba846 => generated 226 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:57.173390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1132f51d-be0b-4f5a-b01a-37f35f5012e1 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:57.218739 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d5cbc1-a300-404f-a81b-e877a63d3c1e None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:57.294486 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:57.300323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1132f51d-be0b-4f5a-b01a-37f35f5012e1 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:57.305802 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:57.342636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9099c573-3e14-44c3-ae9b-ce760a7bf413 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:57.343247 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9099c573-3e14-44c3-ae9b-ce760a7bf413 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:57.343606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9099c573-3e14-44c3-ae9b-ce760a7bf413 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:57.343816 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9099c573-3e14-44c3-ae9b-ce760a7bf413 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:57.344542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9099c573-3e14-44c3-ae9b-ce760a7bf413 None admin] RBAC: Adding query filter params (group.id=496de8c8aebb422fbcae2f9e32bba846, role.id=bb354a7b796d467d91ff4b184011796e, scope.project.id=5847498d7fa84446a3389fbb99498368) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:57.344849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9099c573-3e14-44c3-ae9b-ce760a7bf413 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:57.346560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9099c573-3e14-44c3-ae9b-ce760a7bf413 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:57.348534 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d5cbc1-a300-404f-a81b-e877a63d3c1e None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:57.353751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 167/324] 10.222.0.36 () {56 vars in 1376 bytes} [Mon Oct 31 14:43:57 2022] GET /identity/v3/role_assignments?group.id=496de8c8aebb422fbcae2f9e32bba846&scope.project.id=5847498d7fa84446a3389fbb99498368&role.id=bb354a7b796d467d91ff4b184011796e => generated 611 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:57.354758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9099c573-3e14-44c3-ae9b-ce760a7bf413 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:57.399663 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e8057e-57ff-4a29-b715-81c96ccff604 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:57.400653 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e8057e-57ff-4a29-b715-81c96ccff604 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:57.400939 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e8057e-57ff-4a29-b715-81c96ccff604 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:57.401211 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e8057e-57ff-4a29-b715-81c96ccff604 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:57.414383 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e8057e-57ff-4a29-b715-81c96ccff604 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:57.420712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-f4e8057e-57ff-4a29-b715-81c96ccff604 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:43:57.420712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:57.420712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:57.420712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:57.420712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:57.420712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:57.420712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:57.420712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:57.420712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:57.420712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:57.420712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:57.420712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:57.420712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:57.420712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:57.420712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:57.420712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:57.422016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:57.422016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:57.422016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:57.422016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:57.422016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:57.422016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:57.422016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:57.422016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:57.422016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:57.422016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:57.422016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:57.422016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:57.422016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:57.422016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:57.422016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:57.422016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:57.423003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:57.423003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:57.423003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:57.423003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:57.423003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:57.423003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:57.423003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:57.423003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:57.423003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:57.423003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:57.423003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:43:57.423003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:43:57.423003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 158/325] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:43:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:57.427201 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e8057e-57ff-4a29-b715-81c96ccff604 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:57.521067 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9099c573-3e14-44c3-ae9b-ce760a7bf413 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:57.564159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e8057e-57ff-4a29-b715-81c96ccff604 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:57.564362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ee1b98-3158-4b7b-bf2c-73c57e073c1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:57.565001 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ee1b98-3158-4b7b-bf2c-73c57e073c1e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:57.565281 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ee1b98-3158-4b7b-bf2c-73c57e073c1e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:57.565557 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ee1b98-3158-4b7b-bf2c-73c57e073c1e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:57.578499 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ee1b98-3158-4b7b-bf2c-73c57e073c1e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:43:57.584051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-91ee1b98-3158-4b7b-bf2c-73c57e073c1e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:43:57.584051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:43:57.584051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:43:57.584051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:43:57.584051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:43:57.584051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:43:57.584051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:43:57.584051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:43:57.584051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:43:57.584051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:43:57.584051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:43:57.584051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:43:57.584051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:43:57.584051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:43:57.584051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:43:57.584051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:43:57.585013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:43:57.585013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:43:57.585013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:43:57.585013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:43:57.585013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:43:57.585013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:43:57.585013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:43:57.585013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:43:57.585013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:43:57.585013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:43:57.585013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:43:57.585013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:43:57.585013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:43:57.585013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:43:57.585013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:43:57.585013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:43:57.585882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:43:57.585882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:43:57.585882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:43:57.585882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:43:57.585882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:43:57.585882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:43:57.585882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:43:57.585882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:43:57.585882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:43:57.585882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:43:57.585882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:43:57.585882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:43:57.585882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 168/326] 10.222.0.36 () {56 vars in 1114 bytes} [Mon Oct 31 14:43:57 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:43:57.591695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ee1b98-3158-4b7b-bf2c-73c57e073c1e None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:57.600256 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:57.611936 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c728da35-d5c3-48f1-b012-e2c0df0c3386 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:57.612549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c728da35-d5c3-48f1-b012-e2c0df0c3386 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:57.612833 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c728da35-d5c3-48f1-b012-e2c0df0c3386 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:57.613108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c728da35-d5c3-48f1-b012-e2c0df0c3386 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:57.613788 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c728da35-d5c3-48f1-b012-e2c0df0c3386 None admin] RBAC: Adding query filter params (name=admin) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:57.614157 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c728da35-d5c3-48f1-b012-e2c0df0c3386 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:57.615542 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c728da35-d5c3-48f1-b012-e2c0df0c3386 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:57.623883 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:57.625719 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 159/327] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:43:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:57.631455 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c728da35-d5c3-48f1-b012-e2c0df0c3386 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:57.721906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ee1b98-3158-4b7b-bf2c-73c57e073c1e None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:57.753718 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c728da35-d5c3-48f1-b012-e2c0df0c3386 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:57.765248 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edda5cd-5dec-47ef-9960-e827d5d9bc33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:57.766201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edda5cd-5dec-47ef-9960-e827d5d9bc33 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:57.766772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edda5cd-5dec-47ef-9960-e827d5d9bc33 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:57.767055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edda5cd-5dec-47ef-9960-e827d5d9bc33 None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:57.767812 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edda5cd-5dec-47ef-9960-e827d5d9bc33 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:57.768124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edda5cd-5dec-47ef-9960-e827d5d9bc33 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:57.769499 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edda5cd-5dec-47ef-9960-e827d5d9bc33 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:57.781829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 169/328] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:43:57 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:57.789602 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edda5cd-5dec-47ef-9960-e827d5d9bc33 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:57.794398 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b5f7d1-42fa-46be-93ad-c5958e3eeae0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:57.794398 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b5f7d1-42fa-46be-93ad-c5958e3eeae0 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:57.794398 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b5f7d1-42fa-46be-93ad-c5958e3eeae0 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:57.794398 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b5f7d1-42fa-46be-93ad-c5958e3eeae0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:57.865422 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:57.879765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b5f7d1-42fa-46be-93ad-c5958e3eeae0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:57.881233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b5f7d1-42fa-46be-93ad-c5958e3eeae0 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:57.891524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:57.909121 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edda5cd-5dec-47ef-9960-e827d5d9bc33 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:57.955633 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e0a586-fcb3-4fe3-bd41-311a15f72fe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:57.956508 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e0a586-fcb3-4fe3-bd41-311a15f72fe1 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:57.956791 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e0a586-fcb3-4fe3-bd41-311a15f72fe1 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:57.957061 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e0a586-fcb3-4fe3-bd41-311a15f72fe1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:58.002530 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 160/329] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:43:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 31 14:43:58.007358 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b5f7d1-42fa-46be-93ad-c5958e3eeae0 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:58.041920 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e0a586-fcb3-4fe3-bd41-311a15f72fe1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:58.043420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e0a586-fcb3-4fe3-bd41-311a15f72fe1 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:58.111441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:58.126881 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b5f7d1-42fa-46be-93ad-c5958e3eeae0 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:58.151756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:58.168707 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1293d205-ba0e-43d7-910b-1ebd33e1ab73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:58.169336 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1293d205-ba0e-43d7-910b-1ebd33e1ab73 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:58.169617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1293d205-ba0e-43d7-910b-1ebd33e1ab73 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:58.169913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1293d205-ba0e-43d7-910b-1ebd33e1ab73 None admin] PATH_INFO: `/v3/projects/e18b2b57837347e48db3adc0214764b8` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:58.171015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 170/330] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:43:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 31 14:43:58.171386 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1293d205-ba0e-43d7-910b-1ebd33e1ab73 None admin] RBAC: Authorizing `identity:get_project(project_id=e18b2b57837347e48db3adc0214764b8)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:58.172932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1293d205-ba0e-43d7-910b-1ebd33e1ab73 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:58.174577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 161/331] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:58 2022] GET /identity/v3/projects/e18b2b57837347e48db3adc0214764b8 => generated 344 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:58.177652 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e0a586-fcb3-4fe3-bd41-311a15f72fe1 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:58.179681 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1293d205-ba0e-43d7-910b-1ebd33e1ab73 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:58.276675 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e0a586-fcb3-4fe3-bd41-311a15f72fe1 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:58.278349 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1293d205-ba0e-43d7-910b-1ebd33e1ab73 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:58.310198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495109a2-d267-41e6-929a-13c5f71da253 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:58.310659 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07159f82-1d05-47e5-ae5b-524a9e954bd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:58.311034 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495109a2-d267-41e6-929a-13c5f71da253 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:58.311291 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07159f82-1d05-47e5-ae5b-524a9e954bd8 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:58.311390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495109a2-d267-41e6-929a-13c5f71da253 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:58.311586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495109a2-d267-41e6-929a-13c5f71da253 None admin] PATH_INFO: `/v3/projects/5847498d7fa84446a3389fbb99498368` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:58.311694 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07159f82-1d05-47e5-ae5b-524a9e954bd8 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:58.311859 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07159f82-1d05-47e5-ae5b-524a9e954bd8 None admin] PATH_INFO: `/v3/groups/28b8df505ef941b0a9b8f50e79f5986c` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:58.312705 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495109a2-d267-41e6-929a-13c5f71da253 None admin] RBAC: Authorizing `identity:get_project(project_id=5847498d7fa84446a3389fbb99498368)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:58.312938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07159f82-1d05-47e5-ae5b-524a9e954bd8 None admin] RBAC: Authorizing `identity:get_group(group_id=28b8df505ef941b0a9b8f50e79f5986c)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:58.314129 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495109a2-d267-41e6-929a-13c5f71da253 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:58.314342 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07159f82-1d05-47e5-ae5b-524a9e954bd8 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:58.315770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 171/332] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:43:58 2022] GET /identity/v3/projects/5847498d7fa84446a3389fbb99498368 => generated 302 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:58.320604 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 162/333] 10.222.0.36 () {56 vars in 1180 bytes} [Mon Oct 31 14:43:58 2022] GET /identity/v3/groups/28b8df505ef941b0a9b8f50e79f5986c => generated 229 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:58.321081 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495109a2-d267-41e6-929a-13c5f71da253 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:58.321294 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07159f82-1d05-47e5-ae5b-524a9e954bd8 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:58.346997 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:58.376340 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:58.420771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495109a2-d267-41e6-929a-13c5f71da253 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:58.421589 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07159f82-1d05-47e5-ae5b-524a9e954bd8 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:58.454708 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abd0060-4c9c-4650-aab6-53e3a9d7dc60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:58.455216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8f45ad-bf9f-450b-bd04-e075566a4795 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:58.455328 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abd0060-4c9c-4650-aab6-53e3a9d7dc60 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:58.455590 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abd0060-4c9c-4650-aab6-53e3a9d7dc60 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:58.455870 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abd0060-4c9c-4650-aab6-53e3a9d7dc60 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:58.455966 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8f45ad-bf9f-450b-bd04-e075566a4795 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:58.456145 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8f45ad-bf9f-450b-bd04-e075566a4795 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:58.456417 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8f45ad-bf9f-450b-bd04-e075566a4795 None admin] PATH_INFO: `/v3/groups/496de8c8aebb422fbcae2f9e32bba846` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:58.456592 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abd0060-4c9c-4650-aab6-53e3a9d7dc60 None admin] RBAC: Adding query filter params (group.id=28b8df505ef941b0a9b8f50e79f5986c, role.id=57572a37675d4c418e13794a7b8f3a5d, scope.project.id=e18b2b57837347e48db3adc0214764b8) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:58.456899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abd0060-4c9c-4650-aab6-53e3a9d7dc60 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:58.457507 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8f45ad-bf9f-450b-bd04-e075566a4795 None admin] RBAC: Authorizing `identity:get_group(group_id=496de8c8aebb422fbcae2f9e32bba846)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:58.458321 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abd0060-4c9c-4650-aab6-53e3a9d7dc60 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:58.458917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8f45ad-bf9f-450b-bd04-e075566a4795 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:58.460506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 172/334] 10.222.0.36 () {56 vars in 1180 bytes} [Mon Oct 31 14:43:58 2022] GET /identity/v3/groups/496de8c8aebb422fbcae2f9e32bba846 => generated 226 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:43:58.464080 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 163/335] 10.222.0.36 () {56 vars in 1376 bytes} [Mon Oct 31 14:43:58 2022] GET /identity/v3/role_assignments?group.id=28b8df505ef941b0a9b8f50e79f5986c&scope.project.id=e18b2b57837347e48db3adc0214764b8&role.id=57572a37675d4c418e13794a7b8f3a5d => generated 611 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:43:58.466541 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8f45ad-bf9f-450b-bd04-e075566a4795 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:43:58.574976 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8f45ad-bf9f-450b-bd04-e075566a4795 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:43:58.599854 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:43:58.612851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f80811-9e68-4132-b2ab-f165225595a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:43:58.613528 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f80811-9e68-4132-b2ab-f165225595a3 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:43:58.613811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f80811-9e68-4132-b2ab-f165225595a3 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:43:58.614112 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f80811-9e68-4132-b2ab-f165225595a3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:43:58.614859 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f80811-9e68-4132-b2ab-f165225595a3 None admin] RBAC: Adding query filter params (group.id=496de8c8aebb422fbcae2f9e32bba846, role.id=addf152a48404c27b7d71386e81b8f76, scope.project.id=5847498d7fa84446a3389fbb99498368) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:43:58.615171 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f80811-9e68-4132-b2ab-f165225595a3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:43:58.616663 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f80811-9e68-4132-b2ab-f165225595a3 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:43:58.623311 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 173/336] 10.222.0.36 () {56 vars in 1376 bytes} [Mon Oct 31 14:43:58 2022] GET /identity/v3/role_assignments?group.id=496de8c8aebb422fbcae2f9e32bba846&scope.project.id=5847498d7fa84446a3389fbb99498368&role.id=addf152a48404c27b7d71386e81b8f76 => generated 611 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:01.480849 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30de73f5-ae22-4845-974d-6c51960a6690 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:01.481399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30de73f5-ae22-4845-974d-6c51960a6690 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:01.481399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30de73f5-ae22-4845-974d-6c51960a6690 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:01.483500 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 164/337] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:01.497704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15254f83-0c87-4ef7-8377-227e9c0d077b None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:01.498983 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15254f83-0c87-4ef7-8377-227e9c0d077b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:01.499470 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15254f83-0c87-4ef7-8377-227e9c0d077b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:01.529239 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-15254f83-0c87-4ef7-8377-227e9c0d077b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:01.530824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-15254f83-0c87-4ef7-8377-227e9c0d077b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:01.628617 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 174/338] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:44:01.635755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4400de9-3161-4c97-a289-3cab293085b4 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:01.636038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4400de9-3161-4c97-a289-3cab293085b4 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:01.636314 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4400de9-3161-4c97-a289-3cab293085b4 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:01.637863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 165/339] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:01.643179 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fd0239-33c8-4640-b7a8-22cecc5c67f8 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:01.643513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fd0239-33c8-4640-b7a8-22cecc5c67f8 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:01.643823 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fd0239-33c8-4640-b7a8-22cecc5c67f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:01.664090 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-64fd0239-33c8-4640-b7a8-22cecc5c67f8 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:01.665616 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-64fd0239-33c8-4640-b7a8-22cecc5c67f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:01.748598 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 175/340] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:44:01.937376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9cd952-84cc-47de-82ef-4024918b8678 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:01.938062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9cd952-84cc-47de-82ef-4024918b8678 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:01.938544 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9cd952-84cc-47de-82ef-4024918b8678 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:01.940370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 166/341] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:01.952512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5fa886-2192-4416-9171-f45586e48ebd None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:01.953145 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5fa886-2192-4416-9171-f45586e48ebd None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:01.953621 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5fa886-2192-4416-9171-f45586e48ebd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:01.976204 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-ee5fa886-2192-4416-9171-f45586e48ebd None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:01.977758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ee5fa886-2192-4416-9171-f45586e48ebd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:02.023050 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9cd952-84cc-47de-82ef-4024918b8678 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:02.033651 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:02.061389 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 176/342] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:44:02.138096 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5fa886-2192-4416-9171-f45586e48ebd None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:02.164484 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:02.189812 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9cd952-84cc-47de-82ef-4024918b8678 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:02.255461 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949bf509-494b-4b5a-a43e-811c164a399b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:02.256502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949bf509-494b-4b5a-a43e-811c164a399b None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:02.257000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949bf509-494b-4b5a-a43e-811c164a399b None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:02.257481 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949bf509-494b-4b5a-a43e-811c164a399b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:02.272657 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949bf509-494b-4b5a-a43e-811c164a399b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:02.274432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949bf509-494b-4b5a-a43e-811c164a399b None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:02.299767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-949bf509-494b-4b5a-a43e-811c164a399b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:02.299767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:02.299767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:02.299767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:02.299767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:02.299767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:02.299767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:02.299767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:02.299767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:02.299767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:02.299767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:02.299767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:02.299767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:02.299767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:02.299767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:02.299767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:02.300745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:02.300745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:02.300745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:02.300745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:02.300745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:02.300745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:02.300745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:02.300745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:02.300745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:02.300745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:02.300745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:02.300745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:02.300745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:02.300745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:02.300745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:02.300745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:02.300745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:02.301566 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:02.301566 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:02.301566 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:02.304018 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 167/343] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:44:02.310331 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949bf509-494b-4b5a-a43e-811c164a399b None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:02.346519 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5fa886-2192-4416-9171-f45586e48ebd None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:02.405962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a92d1c-90c2-4f1a-8043-a6251153868f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:02.406886 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a92d1c-90c2-4f1a-8043-a6251153868f None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:02.407168 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a92d1c-90c2-4f1a-8043-a6251153868f None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:02.407445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a92d1c-90c2-4f1a-8043-a6251153868f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:02.417913 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a92d1c-90c2-4f1a-8043-a6251153868f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:02.419362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a92d1c-90c2-4f1a-8043-a6251153868f None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:02.437440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-34a92d1c-90c2-4f1a-8043-a6251153868f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:02.437440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:02.437440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:02.437440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:02.437440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:02.437440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:02.437440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:02.437440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:02.437440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:02.437440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:02.437440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:02.437440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:02.437440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:02.437440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:02.437440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:02.437440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:02.438411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:02.438411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:02.438411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:02.438411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:02.438411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:02.438411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:02.438411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:02.438411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:02.438411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:02.438411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:02.438411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:02.438411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:02.438411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:02.438411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:02.438411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:02.438411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:02.438411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:02.439722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:02.439722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:02.439722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:02.439722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 177/344] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:44:02.440115 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a92d1c-90c2-4f1a-8043-a6251153868f None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:02.449968 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949bf509-494b-4b5a-a43e-811c164a399b None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:02.470246 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:02.492215 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb20419-b328-4b91-8339-18b6252b97a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:02.492817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb20419-b328-4b91-8339-18b6252b97a4 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:02.493094 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb20419-b328-4b91-8339-18b6252b97a4 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:02.493367 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb20419-b328-4b91-8339-18b6252b97a4 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:02.494006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb20419-b328-4b91-8339-18b6252b97a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:02.494335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb20419-b328-4b91-8339-18b6252b97a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:02.495712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb20419-b328-4b91-8339-18b6252b97a4 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:02.504141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:02.511704 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 168/345] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:02.512770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb20419-b328-4b91-8339-18b6252b97a4 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:02.555219 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a92d1c-90c2-4f1a-8043-a6251153868f None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:02.591078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ae6e57-840e-47b1-8e22-392c3e5832c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:02.592322 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ae6e57-840e-47b1-8e22-392c3e5832c5 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:02.592638 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ae6e57-840e-47b1-8e22-392c3e5832c5 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:02.593058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ae6e57-840e-47b1-8e22-392c3e5832c5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:02.606266 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ae6e57-840e-47b1-8e22-392c3e5832c5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:02.608549 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ae6e57-840e-47b1-8e22-392c3e5832c5 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:02.621206 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb20419-b328-4b91-8339-18b6252b97a4 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:02.624978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-f1ae6e57-840e-47b1-8e22-392c3e5832c5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:02.624978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:02.624978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:02.624978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:02.624978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:02.624978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:02.624978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:02.624978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:02.624978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:02.624978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:02.624978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:02.624978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:02.624978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:02.624978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:02.624978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:02.624978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:02.625894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:02.625894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:02.625894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:02.625894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:02.625894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:02.625894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:02.625894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:02.625894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:02.625894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:02.625894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:02.625894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:02.625894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:02.625894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:02.625894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:02.625894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:02.625894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:02.625894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:02.632092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:02.632092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:02.632092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:02.632092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 178/346] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:02.632092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cb01a7-6103-45bb-a14e-1ec0f51be9b1 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:02.632092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cb01a7-6103-45bb-a14e-1ec0f51be9b1 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:02.632092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cb01a7-6103-45bb-a14e-1ec0f51be9b1 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:02.632092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 179/347] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:02.632092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cb01a7-6103-45bb-a14e-1ec0f51be9b1 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:02.662952 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea78e302-be93-4af8-92c1-e4f6df4ab05d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:02.663835 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea78e302-be93-4af8-92c1-e4f6df4ab05d None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:02.664475 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea78e302-be93-4af8-92c1-e4f6df4ab05d None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:02.664769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea78e302-be93-4af8-92c1-e4f6df4ab05d None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:02.665814 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea78e302-be93-4af8-92c1-e4f6df4ab05d None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:02.666127 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea78e302-be93-4af8-92c1-e4f6df4ab05d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:02.667581 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea78e302-be93-4af8-92c1-e4f6df4ab05d None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:02.683067 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 169/348] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:02.684383 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea78e302-be93-4af8-92c1-e4f6df4ab05d None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:02.726438 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:02.748080 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:02.750162 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cb01a7-6103-45bb-a14e-1ec0f51be9b1 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:02.788980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d3a037-a787-458c-aa6b-50b9ed024c86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:02.789524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d3a037-a787-458c-aa6b-50b9ed024c86 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:02.789802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d3a037-a787-458c-aa6b-50b9ed024c86 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:02.790076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d3a037-a787-458c-aa6b-50b9ed024c86 None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:02.790725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d3a037-a787-458c-aa6b-50b9ed024c86 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:02.792186 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d3a037-a787-458c-aa6b-50b9ed024c86 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:02.792536 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea78e302-be93-4af8-92c1-e4f6df4ab05d None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:02.819746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 180/349] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:44:02 2022] POST /identity/v3/users => generated 246 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:02.821800 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbddb82d-c8f8-48a0-9fa0-f20bffa2ec51 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:02.822101 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbddb82d-c8f8-48a0-9fa0-f20bffa2ec51 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:02.822378 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbddb82d-c8f8-48a0-9fa0-f20bffa2ec51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:02.831171 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bc1410-6075-4f92-b4c6-603dd2e16580 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:02.831830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bc1410-6075-4f92-b4c6-603dd2e16580 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:02.832112 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bc1410-6075-4f92-b4c6-603dd2e16580 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:02.832385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bc1410-6075-4f92-b4c6-603dd2e16580 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:02.833050 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8bc1410-6075-4f92-b4c6-603dd2e16580 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:02.833358 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8bc1410-6075-4f92-b4c6-603dd2e16580 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:02.834773 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8bc1410-6075-4f92-b4c6-603dd2e16580 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:02.843442 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-fbddb82d-c8f8-48a0-9fa0-f20bffa2ec51 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:02.844729 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-fbddb82d-c8f8-48a0-9fa0-f20bffa2ec51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:02.848678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 170/350] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:02.849930 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bc1410-6075-4f92-b4c6-603dd2e16580 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:02.925925 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 181/351] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:44:02.927337 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21faa800-5835-4eb2-81e8-4b4c69202644 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:02.927725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21faa800-5835-4eb2-81e8-4b4c69202644 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:02.928062 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21faa800-5835-4eb2-81e8-4b4c69202644 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:02.931618 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 182/352] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:02.931618 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21faa800-5835-4eb2-81e8-4b4c69202644 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:02.975018 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bc1410-6075-4f92-b4c6-603dd2e16580 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:03.003587 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:03.012394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:03.022995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5d6c14-33d2-4a54-bb48-687125f22924 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:03.023632 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5d6c14-33d2-4a54-bb48-687125f22924 None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:03.023915 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5d6c14-33d2-4a54-bb48-687125f22924 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:03.024186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5d6c14-33d2-4a54-bb48-687125f22924 None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:03.025281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5d6c14-33d2-4a54-bb48-687125f22924 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:03.027095 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5d6c14-33d2-4a54-bb48-687125f22924 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:03.054050 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 171/353] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:44:02 2022] POST /identity/v3/users => generated 248 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:03.056617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ce768b-a2cd-4794-bdb6-e847f77954cb None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:03.056929 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ce768b-a2cd-4794-bdb6-e847f77954cb None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:03.057213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ce768b-a2cd-4794-bdb6-e847f77954cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:03.074934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21faa800-5835-4eb2-81e8-4b4c69202644 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:03.080143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-c4ce768b-a2cd-4794-bdb6-e847f77954cb None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:03.080601 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-c4ce768b-a2cd-4794-bdb6-e847f77954cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:03.118158 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b3e92c-7bbc-477c-acfd-f4b2b5cdcc9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:03.118778 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b3e92c-7bbc-477c-acfd-f4b2b5cdcc9d None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:03.119051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b3e92c-7bbc-477c-acfd-f4b2b5cdcc9d None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:03.119622 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b3e92c-7bbc-477c-acfd-f4b2b5cdcc9d None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:03.121240 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b3e92c-7bbc-477c-acfd-f4b2b5cdcc9d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:03.123244 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b3e92c-7bbc-477c-acfd-f4b2b5cdcc9d None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:03.154879 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 183/354] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:44:02 2022] POST /identity/v3/users => generated 248 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:03.161679 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 172/355] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:44:03.215448 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b3e92c-7bbc-477c-acfd-f4b2b5cdcc9d None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:03.375154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b3e92c-7bbc-477c-acfd-f4b2b5cdcc9d None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:03.407139 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:03.422692 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a937b0a6-0520-4e5a-9101-65ece86e2216 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:03.423375 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a937b0a6-0520-4e5a-9101-65ece86e2216 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:03.423772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a937b0a6-0520-4e5a-9101-65ece86e2216 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:03.424044 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a937b0a6-0520-4e5a-9101-65ece86e2216 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:03.436545 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a937b0a6-0520-4e5a-9101-65ece86e2216 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:03.437993 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a937b0a6-0520-4e5a-9101-65ece86e2216 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:03.452406 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ce768b-a2cd-4794-bdb6-e847f77954cb None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:03.453533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-a937b0a6-0520-4e5a-9101-65ece86e2216 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:03.453533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:03.453533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:03.453533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:03.453533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:03.453533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:03.453533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:03.453533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:03.453533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:03.453533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:03.453533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:03.453533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:03.453533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:03.453533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:03.453533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:03.453533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:03.454409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:03.454409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:03.454409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:03.454409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:03.454409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:03.454409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:03.454409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:03.454409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:03.454409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:03.454409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:03.454409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:03.454409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:03.454409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:03.454409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:03.454409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:03.454409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:03.454409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:03.455220 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:03.455220 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:03.455220 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:03.455220 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 184/356] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:03.460694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a937b0a6-0520-4e5a-9101-65ece86e2216 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:03.582609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a937b0a6-0520-4e5a-9101-65ece86e2216 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:03.620593 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd64443-23d0-48fd-9939-3d086e66ed8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:03.621203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd64443-23d0-48fd-9939-3d086e66ed8a None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:03.621480 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd64443-23d0-48fd-9939-3d086e66ed8a None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:03.621758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd64443-23d0-48fd-9939-3d086e66ed8a None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:03.622406 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd64443-23d0-48fd-9939-3d086e66ed8a None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:03.622780 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd64443-23d0-48fd-9939-3d086e66ed8a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:03.624279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd64443-23d0-48fd-9939-3d086e66ed8a None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:03.637931 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 185/357] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:03.645720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ce768b-a2cd-4794-bdb6-e847f77954cb None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:03.650209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd64443-23d0-48fd-9939-3d086e66ed8a None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:03.663941 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:03.689254 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:03.757649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b587f1-e2d2-4279-9c60-723e1f413a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:03.758269 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b587f1-e2d2-4279-9c60-723e1f413a35 None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:03.758548 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b587f1-e2d2-4279-9c60-723e1f413a35 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:03.758821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b587f1-e2d2-4279-9c60-723e1f413a35 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:03.759492 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b587f1-e2d2-4279-9c60-723e1f413a35 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:03.760510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd64443-23d0-48fd-9939-3d086e66ed8a None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:03.760880 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b587f1-e2d2-4279-9c60-723e1f413a35 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:03.788602 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.sql.core [None req-97b587f1-e2d2-4279-9c60-723e1f413a35 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 31 14:44:03.788602 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 31 14:44:03.788602 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [parameters: {'id': 'a71d730862ca4af196a3a7dcc011e165', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 31 14:44:03.788602 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=96564) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 31 14:44:03.798155 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ec3da9-58ff-4e35-a7b1-09f3708d125c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:03.798774 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ec3da9-58ff-4e35-a7b1-09f3708d125c None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:03.799055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ec3da9-58ff-4e35-a7b1-09f3708d125c None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:03.799331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ec3da9-58ff-4e35-a7b1-09f3708d125c None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:03.800023 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ec3da9-58ff-4e35-a7b1-09f3708d125c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:03.801602 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ec3da9-58ff-4e35-a7b1-09f3708d125c None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:03.823262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-97b587f1-e2d2-4279-9c60-723e1f413a35 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 31 14:44:03.823262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:03.823262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 31 14:44:03.823262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 31 14:44:03.823262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 31 14:44:03.823262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 31 14:44:03.823262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 31 14:44:03.823262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application result = self._query(query) Oct 31 14:44:03.823262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 31 14:44:03.823262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application conn.query(q) Oct 31 14:44:03.823262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 31 14:44:03.823262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 31 14:44:03.823262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 31 14:44:03.823262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application result.read() Oct 31 14:44:03.823262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return ref.to_dict() Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 31 14:44:03.824216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application next(self.gen) Oct 31 14:44:03.825176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 31 14:44:03.825176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application yield resource Oct 31 14:44:03.825176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 31 14:44:03.825176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application next(self.gen) Oct 31 14:44:03.825176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 31 14:44:03.825176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.session.rollback() Oct 31 14:44:03.825176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 31 14:44:03.825176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.force_reraise() Oct 31 14:44:03.825176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 31 14:44:03.825176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise self.value Oct 31 14:44:03.825176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 31 14:44:03.825176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 31 14:44:03.825176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 31 14:44:03.825176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application session.commit() Oct 31 14:44:03.825176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 31 14:44:03.825176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 31 14:44:03.825176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 31 14:44:03.826565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._prepare_impl() Oct 31 14:44:03.826565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 31 14:44:03.826565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.session.flush() Oct 31 14:44:03.826565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 31 14:44:03.826565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._flush(objects) Oct 31 14:44:03.826565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 31 14:44:03.826565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 31 14:44:03.826565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 31 14:44:03.826565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application compat.raise_( Oct 31 14:44:03.826565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 31 14:44:03.826565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception Oct 31 14:44:03.826565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 31 14:44:03.826565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application flush_context.execute() Oct 31 14:44:03.826565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 31 14:44:03.826565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rec.execute(self) Oct 31 14:44:03.826565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 31 14:44:03.826565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 31 14:44:03.827419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 31 14:44:03.827419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 31 14:44:03.827419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 31 14:44:03.827419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 31 14:44:03.827419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 31 14:44:03.827419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 31 14:44:03.827419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 31 14:44:03.827419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 31 14:44:03.827419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 31 14:44:03.827419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 31 14:44:03.827419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 31 14:44:03.827419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 31 14:44:03.827419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 31 14:44:03.827419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 31 14:44:03.827419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 31 14:44:03.827419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception Oct 31 14:44:03.829995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 31 14:44:03.829995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 31 14:44:03.829995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 31 14:44:03.829995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 31 14:44:03.829995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 31 14:44:03.829995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application result = self._query(query) Oct 31 14:44:03.829995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 31 14:44:03.829995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application conn.query(q) Oct 31 14:44:03.829995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 31 14:44:03.829995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 31 14:44:03.829995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 31 14:44:03.829995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application result.read() Oct 31 14:44:03.829995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 31 14:44:03.829995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 31 14:44:03.829995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 31 14:44:03.829995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 31 14:44:03.831967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 31 14:44:03.831967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 31 14:44:03.831967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 31 14:44:03.831967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 31 14:44:03.831967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 31 14:44:03.831967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 31 14:44:03.831967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [parameters: {'id': 'a71d730862ca4af196a3a7dcc011e165', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 31 14:44:03.831967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 31 14:44:03.831967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:03.831967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:03.831967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:03.831967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:03.831967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:03.831967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:03.831967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:03.831967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:03.834188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:03.834188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:03.834188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:03.834188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:03.834188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:03.834188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:03.834188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 31 14:44:03.834188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 31 14:44:03.834188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:03.834188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:03.834188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 31 14:44:03.834188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 31 14:44:03.834188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 31 14:44:03.834188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 31 14:44:03.834188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 31 14:44:03.834188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:03.836614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 173/358] 10.222.0.36 () {60 vars in 1131 bytes} [Mon Oct 31 14:44:03 2022] POST /identity/v3/roles => generated 144 bytes in 378 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:03.836614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b587f1-e2d2-4279-9c60-723e1f413a35 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:03.844061 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 186/359] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:44:03 2022] POST /identity/v3/users => generated 249 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:03.962044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b587f1-e2d2-4279-9c60-723e1f413a35 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:04.006970 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dedf8a3-3465-4c75-b862-ae898bb8c90e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:04.007657 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dedf8a3-3465-4c75-b862-ae898bb8c90e None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:04.007952 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dedf8a3-3465-4c75-b862-ae898bb8c90e None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:04.008231 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dedf8a3-3465-4c75-b862-ae898bb8c90e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:04.025823 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dedf8a3-3465-4c75-b862-ae898bb8c90e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:04.032102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-1dedf8a3-3465-4c75-b862-ae898bb8c90e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:44:04.032102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:04.032102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:04.032102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:04.032102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:04.032102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:04.032102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:04.032102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:04.032102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:04.032102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:04.032102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:04.032102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:04.032102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:04.032102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:04.032102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:04.032102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:04.034223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:04.034223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:04.034223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:04.034223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:04.034223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:04.034223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:04.034223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:04.034223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:04.034223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:04.034223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:04.034223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:04.034223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:04.034223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:04.034223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:04.034223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:04.034223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:04.036153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:04.036153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:04.036153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:04.036153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:04.036153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:04.036153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:04.036153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:04.036153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:04.036153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:04.036153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:04.036153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 31 14:44:04.036153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:04.036153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 174/360] 10.222.0.36 () {56 vars in 1114 bytes} [Mon Oct 31 14:44:03 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:04.040257 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ec3da9-58ff-4e35-a7b1-09f3708d125c None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:04.163919 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:04.172719 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ec3da9-58ff-4e35-a7b1-09f3708d125c None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:04.208315 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0ddf28-7e06-4b39-a413-9c7ca86aafba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:04.208931 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0ddf28-7e06-4b39-a413-9c7ca86aafba None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:04.209210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0ddf28-7e06-4b39-a413-9c7ca86aafba None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:04.209479 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0ddf28-7e06-4b39-a413-9c7ca86aafba None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:04.210214 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0ddf28-7e06-4b39-a413-9c7ca86aafba None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:04.210516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0ddf28-7e06-4b39-a413-9c7ca86aafba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:04.211980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0ddf28-7e06-4b39-a413-9c7ca86aafba None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:04.221234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 187/361] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:04 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:05.772538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6329c62-db0d-459e-8024-c2369c4c2667 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:05.773181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6329c62-db0d-459e-8024-c2369c4c2667 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:05.773181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6329c62-db0d-459e-8024-c2369c4c2667 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:05.775542 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 175/362] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:05.781464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cf8316-9ccc-4eaa-926b-93fce265e5e2 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:05.782220 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cf8316-9ccc-4eaa-926b-93fce265e5e2 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:05.782220 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cf8316-9ccc-4eaa-926b-93fce265e5e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:05.808265 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-45cf8316-9ccc-4eaa-926b-93fce265e5e2 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:05.809421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-45cf8316-9ccc-4eaa-926b-93fce265e5e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:05.889410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfde0335-e87a-4c72-976b-60dacc0247b1 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:05.889877 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfde0335-e87a-4c72-976b-60dacc0247b1 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:05.890158 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfde0335-e87a-4c72-976b-60dacc0247b1 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:05.891515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 176/363] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:44:05.897380 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2791650a-35f4-45ff-8a3d-499c2851b8e7 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:05.897663 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2791650a-35f4-45ff-8a3d-499c2851b8e7 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:05.897934 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2791650a-35f4-45ff-8a3d-499c2851b8e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:05.917564 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 188/364] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:44:05.920325 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2935d6a0-0eea-4f39-b9d5-1e94bc3ceb2c None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:05.920650 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2935d6a0-0eea-4f39-b9d5-1e94bc3ceb2c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:05.920816 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2935d6a0-0eea-4f39-b9d5-1e94bc3ceb2c None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:05.922301 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 189/365] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:05.923248 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-2791650a-35f4-45ff-8a3d-499c2851b8e7 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:05.924621 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2791650a-35f4-45ff-8a3d-499c2851b8e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:05.927577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d971b386-8f86-41dc-92fd-29bcca0b2592 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:05.927867 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d971b386-8f86-41dc-92fd-29bcca0b2592 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:05.928140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d971b386-8f86-41dc-92fd-29bcca0b2592 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:05.946881 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-d971b386-8f86-41dc-92fd-29bcca0b2592 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:05.948088 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d971b386-8f86-41dc-92fd-29bcca0b2592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:05.969236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:06.018414 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 190/366] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:44:06.052820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 177/367] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:44:06.267992 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d971b386-8f86-41dc-92fd-29bcca0b2592 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:06.283468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:06.354293 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2791650a-35f4-45ff-8a3d-499c2851b8e7 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:06.381254 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d971b386-8f86-41dc-92fd-29bcca0b2592 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:06.418916 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d10188-81ae-4995-8e11-476e4938d2ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:06.421418 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d10188-81ae-4995-8e11-476e4938d2ad None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:06.421418 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d10188-81ae-4995-8e11-476e4938d2ad None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:06.421418 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d10188-81ae-4995-8e11-476e4938d2ad None admin] PATH_INFO: `/v3/roles/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:06.431376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d10188-81ae-4995-8e11-476e4938d2ad None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:06.437811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-f7d10188-81ae-4995-8e11-476e4938d2ad None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:06.437811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:06.437811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:06.437811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:06.437811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:06.437811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:06.437811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:06.437811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:06.437811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:06.437811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:06.437811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:06.437811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:06.437811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:06.437811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:06.437811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:06.437811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:06.439855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:06.439855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:06.439855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:06.439855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:06.439855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:06.439855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:06.439855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:06.439855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:06.439855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:06.439855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:06.439855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:06.439855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:06.439855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:06.439855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:06.439855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:06.439855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:06.441771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:06.441771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:06.441771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:06.441771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:06.441771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:06.441771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:06.441771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:06.441771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:06.441771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:06.441771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:06.441771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:06.441771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:06.441771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 191/368] 10.222.0.36 () {56 vars in 1102 bytes} [Mon Oct 31 14:44:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:06.445859 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d10188-81ae-4995-8e11-476e4938d2ad None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:06.524262 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:06.540907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2791650a-35f4-45ff-8a3d-499c2851b8e7 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:06.559391 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d10188-81ae-4995-8e11-476e4938d2ad None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:06.581615 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a6d056-f251-444e-ac96-42619fad1f5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:06.582385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a6d056-f251-444e-ac96-42619fad1f5a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:06.582759 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a6d056-f251-444e-ac96-42619fad1f5a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:06.583149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a6d056-f251-444e-ac96-42619fad1f5a None admin] PATH_INFO: `/v3/roles/service` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:06.595852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26724bf4-0b9d-4eb8-9994-76c14827424e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:06.596511 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a6d056-f251-444e-ac96-42619fad1f5a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:06.596795 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26724bf4-0b9d-4eb8-9994-76c14827424e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:06.597209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26724bf4-0b9d-4eb8-9994-76c14827424e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:06.597637 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26724bf4-0b9d-4eb8-9994-76c14827424e None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:06.598440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26724bf4-0b9d-4eb8-9994-76c14827424e None admin] RBAC: Adding query filter params (name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:06.598879 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26724bf4-0b9d-4eb8-9994-76c14827424e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:06.600409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26724bf4-0b9d-4eb8-9994-76c14827424e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:06.602353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-a3a6d056-f251-444e-ac96-42619fad1f5a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:06.602353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:06.602353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:06.602353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:06.602353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:06.602353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:06.602353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:06.602353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:06.602353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:06.602353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:06.602353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:06.602353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:06.602353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:06.602353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:06.602353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:06.602353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:06.603444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:06.603444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:06.603444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:06.603444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:06.603444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:06.603444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:06.603444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:06.603444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:06.603444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:06.603444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:06.603444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:06.603444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:06.603444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:06.603444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:06.603444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:06.603444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:06.604305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:06.604305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:06.604305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:06.604305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:06.604305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:06.604305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:06.604305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:06.604305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:06.604305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:06.604305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:06.604305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:06.604305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:06.605190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 178/369] 10.222.0.36 () {56 vars in 1102 bytes} [Mon Oct 31 14:44:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:06.606308 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a6d056-f251-444e-ac96-42619fad1f5a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:06.613399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 192/370] 10.222.0.36 () {56 vars in 1103 bytes} [Mon Oct 31 14:44:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:06.614547 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26724bf4-0b9d-4eb8-9994-76c14827424e None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:06.648391 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:06.727121 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26724bf4-0b9d-4eb8-9994-76c14827424e None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:06.756723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a6d056-f251-444e-ac96-42619fad1f5a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:06.763731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3db73f-9d82-44a5-a08a-74c4afe8ada7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:06.765130 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3db73f-9d82-44a5-a08a-74c4afe8ada7 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:06.765871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3db73f-9d82-44a5-a08a-74c4afe8ada7 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:06.766470 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3db73f-9d82-44a5-a08a-74c4afe8ada7 None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:06.767229 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3db73f-9d82-44a5-a08a-74c4afe8ada7 None admin] RBAC: Adding query filter params (name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:06.767662 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3db73f-9d82-44a5-a08a-74c4afe8ada7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:06.769114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3db73f-9d82-44a5-a08a-74c4afe8ada7 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:06.770751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:06.780036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 193/371] 10.222.0.36 () {56 vars in 1103 bytes} [Mon Oct 31 14:44:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:06.780919 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3db73f-9d82-44a5-a08a-74c4afe8ada7 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:06.805217 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbd22b5-1104-4698-9859-8e4968590023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:06.806670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbd22b5-1104-4698-9859-8e4968590023 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:06.807096 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbd22b5-1104-4698-9859-8e4968590023 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:06.807794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbd22b5-1104-4698-9859-8e4968590023 None admin] PATH_INFO: `/v3/roles/service` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:06.823144 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbd22b5-1104-4698-9859-8e4968590023 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:06.830055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-2cbd22b5-1104-4698-9859-8e4968590023 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:06.830055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:06.830055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:06.830055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:06.830055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:06.830055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:06.830055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:06.830055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:06.830055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:06.830055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:06.830055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:06.830055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:06.830055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:06.830055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:06.830055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:06.830055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:06.830966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:06.830966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:06.830966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:06.830966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:06.830966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:06.830966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:06.830966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:06.830966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:06.830966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:06.830966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:06.830966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:06.830966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:06.830966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:06.830966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:06.830966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:06.830966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:06.831819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:06.831819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:06.831819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:06.831819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:06.831819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:06.831819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:06.831819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:06.831819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:06.831819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:06.831819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:06.831819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:06.831819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:06.831819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 179/372] 10.222.0.36 () {56 vars in 1102 bytes} [Mon Oct 31 14:44:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:06.832545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbd22b5-1104-4698-9859-8e4968590023 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:06.893362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3db73f-9d82-44a5-a08a-74c4afe8ada7 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:06.918491 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:06.932432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e26bcf-7a6b-4f1f-aa01-ce9bc588586a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:06.933069 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e26bcf-7a6b-4f1f-aa01-ce9bc588586a None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:06.933352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e26bcf-7a6b-4f1f-aa01-ce9bc588586a None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:06.933625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e26bcf-7a6b-4f1f-aa01-ce9bc588586a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:06.940108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbd22b5-1104-4698-9859-8e4968590023 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:06.946747 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e26bcf-7a6b-4f1f-aa01-ce9bc588586a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:06.948242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e26bcf-7a6b-4f1f-aa01-ce9bc588586a None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:06.962938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-a7e26bcf-7a6b-4f1f-aa01-ce9bc588586a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:06.962938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:06.962938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:06.962938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:06.962938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:06.962938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:06.962938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:06.962938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:06.962938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:06.962938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:06.962938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:06.962938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:06.962938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:06.962938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:06.962938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:06.962938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:06.963986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:06.963986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:06.963986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:06.963986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:06.963986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:06.963986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:06.963986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:06.963986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:06.963986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:06.963986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:06.963986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:06.963986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:06.963986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:06.963986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:06.963986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:06.963986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:06.963986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:06.964862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:06.964862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:06.964862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:06.964862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 194/373] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:06.965887 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a3e4ef-6a59-49d4-92d7-6960e568dbcc None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:06.966685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a3e4ef-6a59-49d4-92d7-6960e568dbcc None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:06.968372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a3e4ef-6a59-49d4-92d7-6960e568dbcc None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:06.974059 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 195/374] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:06.974059 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a3e4ef-6a59-49d4-92d7-6960e568dbcc None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:06.983047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c6d9be-c0f7-4f65-9674-badfda36ec06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:06.983711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c6d9be-c0f7-4f65-9674-badfda36ec06 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:06.983994 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c6d9be-c0f7-4f65-9674-badfda36ec06 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:06.984262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c6d9be-c0f7-4f65-9674-badfda36ec06 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:06.994717 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c6d9be-c0f7-4f65-9674-badfda36ec06 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:06.996210 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c6d9be-c0f7-4f65-9674-badfda36ec06 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:07.011436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-d7c6d9be-c0f7-4f65-9674-badfda36ec06 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:07.011436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:07.011436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:07.011436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:07.011436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:07.011436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:07.011436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:07.011436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:07.011436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:07.011436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:07.011436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:07.011436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:07.011436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:07.011436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:07.011436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:07.011436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:07.012381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:07.012381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:07.012381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:07.012381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:07.012381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:07.012381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:07.012381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:07.012381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:07.012381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:07.012381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:07.012381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:07.012381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:07.012381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:07.012381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:07.012381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:07.012381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:07.012381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:07.013341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:07.013341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:07.013341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:07.013341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 180/375] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:07.013341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c6d9be-c0f7-4f65-9674-badfda36ec06 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:07.031680 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:07.088483 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a3e4ef-6a59-49d4-92d7-6960e568dbcc None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:07.118283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c6d9be-c0f7-4f65-9674-badfda36ec06 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:07.123544 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148c33d6-65f6-427d-ac70-342189637077 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:07.124172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148c33d6-65f6-427d-ac70-342189637077 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:07.124450 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148c33d6-65f6-427d-ac70-342189637077 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:07.124724 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148c33d6-65f6-427d-ac70-342189637077 None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:07.125377 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148c33d6-65f6-427d-ac70-342189637077 None admin] RBAC: Adding query filter params (name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:07.125680 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148c33d6-65f6-427d-ac70-342189637077 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:07.127038 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148c33d6-65f6-427d-ac70-342189637077 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:07.139469 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 196/376] 10.222.0.36 () {56 vars in 1103 bytes} [Mon Oct 31 14:44:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:07.139469 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99056cf7-7f7c-4053-86ea-f847a374915e None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:07.139469 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99056cf7-7f7c-4053-86ea-f847a374915e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:07.139469 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99056cf7-7f7c-4053-86ea-f847a374915e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:07.155009 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-99056cf7-7f7c-4053-86ea-f847a374915e None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:07.155522 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:07.156229 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-99056cf7-7f7c-4053-86ea-f847a374915e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:07.161605 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10e3749-76d5-4c6c-b012-bda76a82cc8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:07.162473 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10e3749-76d5-4c6c-b012-bda76a82cc8c None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:07.162933 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10e3749-76d5-4c6c-b012-bda76a82cc8c None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:07.163382 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10e3749-76d5-4c6c-b012-bda76a82cc8c None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:07.164360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10e3749-76d5-4c6c-b012-bda76a82cc8c None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:07.164670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10e3749-76d5-4c6c-b012-bda76a82cc8c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:07.166297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10e3749-76d5-4c6c-b012-bda76a82cc8c None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:07.181710 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 181/377] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:07.187429 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10e3749-76d5-4c6c-b012-bda76a82cc8c None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:07.228505 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 197/378] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:44:07.230186 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99056cf7-7f7c-4053-86ea-f847a374915e None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:07.280141 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:07.303118 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10e3749-76d5-4c6c-b012-bda76a82cc8c None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:07.339192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a00036-8db2-436b-b4a8-02cd70998533 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:07.339771 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a00036-8db2-436b-b4a8-02cd70998533 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:07.340097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a00036-8db2-436b-b4a8-02cd70998533 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:07.340374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a00036-8db2-436b-b4a8-02cd70998533 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:07.341012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a00036-8db2-436b-b4a8-02cd70998533 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:07.341344 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a00036-8db2-436b-b4a8-02cd70998533 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:07.342731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a00036-8db2-436b-b4a8-02cd70998533 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:07.350517 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99056cf7-7f7c-4053-86ea-f847a374915e None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:07.355087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 182/379] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:07.355945 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a00036-8db2-436b-b4a8-02cd70998533 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:07.387884 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d03a8f-8c73-427d-9951-8442e136cb29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:07.389514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d03a8f-8c73-427d-9951-8442e136cb29 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:07.389760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d03a8f-8c73-427d-9951-8442e136cb29 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:07.390036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d03a8f-8c73-427d-9951-8442e136cb29 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:07.403141 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d03a8f-8c73-427d-9951-8442e136cb29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:07.405928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d03a8f-8c73-427d-9951-8442e136cb29 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:07.411415 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:07.422344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-86d03a8f-8c73-427d-9951-8442e136cb29 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:07.422344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:07.422344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:07.422344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:07.422344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:07.422344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:07.422344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:07.422344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:07.422344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:07.422344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:07.422344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:07.422344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:07.422344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:07.422344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:07.422344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:07.422344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:07.423358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:07.423358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:07.423358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:07.423358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:07.423358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:07.423358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:07.423358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:07.423358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:07.423358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:07.423358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:07.423358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:07.423358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:07.423358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:07.423358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:07.423358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:07.423358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:07.423358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:07.424237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:07.424237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:07.424237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:07.424237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 198/380] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:07.425252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9fe035-9e2b-43fc-a34b-9f9eac9d4456 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:07.425252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9fe035-9e2b-43fc-a34b-9f9eac9d4456 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:07.425374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9fe035-9e2b-43fc-a34b-9f9eac9d4456 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:07.426575 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 199/381] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:07.431439 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9fe035-9e2b-43fc-a34b-9f9eac9d4456 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:07.470485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a00036-8db2-436b-b4a8-02cd70998533 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:07.507842 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b786df-6bdb-4f2c-b8e5-b8de787ce532 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:07.508462 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b786df-6bdb-4f2c-b8e5-b8de787ce532 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:07.508736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b786df-6bdb-4f2c-b8e5-b8de787ce532 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:07.509002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b786df-6bdb-4f2c-b8e5-b8de787ce532 None admin] PATH_INFO: `/v3/users/nova` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:07.517399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b786df-6bdb-4f2c-b8e5-b8de787ce532 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:07.518886 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b786df-6bdb-4f2c-b8e5-b8de787ce532 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:07.527735 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:07.534794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-b4b786df-6bdb-4f2c-b8e5-b8de787ce532 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 31 14:44:07.534794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:07.534794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:07.534794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:07.534794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:07.534794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:07.534794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:07.534794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:07.534794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:07.534794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:07.534794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:07.534794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:07.534794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:07.534794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:07.534794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:07.534794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:07.535797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:07.536783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:07.536783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:07.536783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:07.536783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:07.536783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:07.536783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:07.536783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:07.536783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:07.536783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 31 14:44:07.536783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:07.536783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 183/382] 10.222.0.36 () {56 vars in 1093 bytes} [Mon Oct 31 14:44:07 2022] GET /identity/v3/users/nova => generated 82 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:07.536783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b786df-6bdb-4f2c-b8e5-b8de787ce532 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:07.542368 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9fe035-9e2b-43fc-a34b-9f9eac9d4456 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:07.576497 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc32e346-4f74-41ff-956f-f7bb5034b105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:07.577118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc32e346-4f74-41ff-956f-f7bb5034b105 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:07.577397 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc32e346-4f74-41ff-956f-f7bb5034b105 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:07.577673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc32e346-4f74-41ff-956f-f7bb5034b105 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:07.586171 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc32e346-4f74-41ff-956f-f7bb5034b105 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:07.587806 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc32e346-4f74-41ff-956f-f7bb5034b105 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:07.601296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-dc32e346-4f74-41ff-956f-f7bb5034b105 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 31 14:44:07.601296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:07.601296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:07.601296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:07.601296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:07.601296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:07.601296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:07.601296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:07.601296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:07.601296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:07.601296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:07.601296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:07.601296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:07.601296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:07.601296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:07.601296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:07.602366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:07.603468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:07.603468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:07.603468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:07.603468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:07.603468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:07.603468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:07.603468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:07.603468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:07.603468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 31 14:44:07.603468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:07.605604 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 200/383] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:44:07 2022] GET /identity/v3/users/cinder => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:07.606820 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73240eac-135f-4d80-9a29-9d38422514a3 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:07.607478 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73240eac-135f-4d80-9a29-9d38422514a3 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:07.607769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73240eac-135f-4d80-9a29-9d38422514a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:07.626385 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-73240eac-135f-4d80-9a29-9d38422514a3 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:07.627595 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-73240eac-135f-4d80-9a29-9d38422514a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:07.643327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b786df-6bdb-4f2c-b8e5-b8de787ce532 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:07.656792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:07.678283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea61a78-1965-43dc-8631-e341facef982 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:07.678909 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea61a78-1965-43dc-8631-e341facef982 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:07.679192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea61a78-1965-43dc-8631-e341facef982 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:07.679469 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea61a78-1965-43dc-8631-e341facef982 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:07.680145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea61a78-1965-43dc-8631-e341facef982 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:07.680457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea61a78-1965-43dc-8631-e341facef982 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:07.681832 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea61a78-1965-43dc-8631-e341facef982 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:07.698212 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 184/384] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:07.698212 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea61a78-1965-43dc-8631-e341facef982 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:07.715587 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 201/385] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:44:07.715587 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73240eac-135f-4d80-9a29-9d38422514a3 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:07.800549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea61a78-1965-43dc-8631-e341facef982 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:07.801451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:07.822190 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73240eac-135f-4d80-9a29-9d38422514a3 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:07.835652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c426d119-3a5b-43bd-80c5-018236485eca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:07.836331 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c426d119-3a5b-43bd-80c5-018236485eca None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:07.836668 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c426d119-3a5b-43bd-80c5-018236485eca None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:07.836944 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c426d119-3a5b-43bd-80c5-018236485eca None admin] PATH_INFO: `/v3/roles/service` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:07.847725 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c426d119-3a5b-43bd-80c5-018236485eca None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:07.852474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-c426d119-3a5b-43bd-80c5-018236485eca None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:07.852474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:07.852474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:07.852474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:07.852474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:07.852474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:07.852474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:07.852474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:07.852474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:07.852474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:07.852474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:07.852474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:07.852474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:07.852474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:07.852474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:07.852474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:07.853359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:07.853359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:07.853359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:07.853359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:07.853359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:07.853359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:07.853359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:07.853359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:07.853359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:07.853359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:07.853359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:07.853359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:07.853359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:07.853359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:07.853359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:07.853359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:07.854136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:07.854136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:07.854136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:07.854136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:07.854136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:07.854136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:07.854136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:07.854136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:07.854136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:07.854136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:07.854136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:07.854136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:07.854136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 185/386] 10.222.0.36 () {56 vars in 1102 bytes} [Mon Oct 31 14:44:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:07.854826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c426d119-3a5b-43bd-80c5-018236485eca None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:07.866734 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4e71d5-970e-4f8a-9002-9013021857c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:07.867356 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4e71d5-970e-4f8a-9002-9013021857c6 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:07.867668 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4e71d5-970e-4f8a-9002-9013021857c6 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:07.867945 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4e71d5-970e-4f8a-9002-9013021857c6 None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:07.868675 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4e71d5-970e-4f8a-9002-9013021857c6 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:07.868983 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4e71d5-970e-4f8a-9002-9013021857c6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:07.870446 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4e71d5-970e-4f8a-9002-9013021857c6 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:07.888840 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:07.911200 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 202/387] 10.222.0.36 () {56 vars in 1133 bytes} [Mon Oct 31 14:44:07 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:07.911963 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4e71d5-970e-4f8a-9002-9013021857c6 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:07.969550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c426d119-3a5b-43bd-80c5-018236485eca None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:08.005814 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57967b93-e99f-4c7e-882a-3cf4bc5d0df3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:08.006411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57967b93-e99f-4c7e-882a-3cf4bc5d0df3 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:08.006694 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57967b93-e99f-4c7e-882a-3cf4bc5d0df3 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:08.006967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57967b93-e99f-4c7e-882a-3cf4bc5d0df3 None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:08.007733 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57967b93-e99f-4c7e-882a-3cf4bc5d0df3 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:08.008042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57967b93-e99f-4c7e-882a-3cf4bc5d0df3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:08.009478 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57967b93-e99f-4c7e-882a-3cf4bc5d0df3 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:08.017240 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4e71d5-970e-4f8a-9002-9013021857c6 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:08.047394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 186/388] 10.222.0.36 () {56 vars in 1137 bytes} [Mon Oct 31 14:44:07 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:08.048077 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57967b93-e99f-4c7e-882a-3cf4bc5d0df3 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:08.051986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d2bbdf-d261-417c-af5a-f920da3aa57c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:08.052643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d2bbdf-d261-417c-af5a-f920da3aa57c None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:08.052924 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d2bbdf-d261-417c-af5a-f920da3aa57c None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:08.053200 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d2bbdf-d261-417c-af5a-f920da3aa57c None admin] PATH_INFO: `/v3/users/glance` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:08.060441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d2bbdf-d261-417c-af5a-f920da3aa57c None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:08.061896 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d2bbdf-d261-417c-af5a-f920da3aa57c None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:08.064457 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:08.074180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-70d2bbdf-d261-417c-af5a-f920da3aa57c None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 31 14:44:08.074180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:08.074180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:08.074180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:08.074180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:08.074180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:08.074180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:08.074180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:08.074180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:08.074180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:08.074180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:08.074180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:08.074180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:08.074180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:08.074180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:08.074180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:08.075126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:08.076779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:08.076779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:08.076779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:08.076779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:08.076779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:08.076779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:08.076779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:08.076779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:08.076779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 31 14:44:08.076779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:08.076779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 203/389] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:44:07 2022] GET /identity/v3/users/glance => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:08.076779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d2bbdf-d261-417c-af5a-f920da3aa57c None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:08.145678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:08.156044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57967b93-e99f-4c7e-882a-3cf4bc5d0df3 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:08.177967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d2bbdf-d261-417c-af5a-f920da3aa57c None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:08.189172 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb94179-f575-4311-a686-8f32dda1768a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:08.189774 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb94179-f575-4311-a686-8f32dda1768a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:08.190056 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb94179-f575-4311-a686-8f32dda1768a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:08.190327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb94179-f575-4311-a686-8f32dda1768a None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:08.190991 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb94179-f575-4311-a686-8f32dda1768a None admin] RBAC: Adding query filter params (name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:08.191295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb94179-f575-4311-a686-8f32dda1768a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:08.192713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb94179-f575-4311-a686-8f32dda1768a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:08.201271 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 187/390] 10.222.0.36 () {56 vars in 1103 bytes} [Mon Oct 31 14:44:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:08.202067 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb94179-f575-4311-a686-8f32dda1768a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:08.216776 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4c495a-8063-4e53-9da4-f933a4183ddc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:08.217412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4c495a-8063-4e53-9da4-f933a4183ddc None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:08.217695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4c495a-8063-4e53-9da4-f933a4183ddc None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:08.217968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4c495a-8063-4e53-9da4-f933a4183ddc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:08.228780 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4c495a-8063-4e53-9da4-f933a4183ddc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:08.230232 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4c495a-8063-4e53-9da4-f933a4183ddc None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:08.244404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-fa4c495a-8063-4e53-9da4-f933a4183ddc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:08.244404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:08.244404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:08.244404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:08.244404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:08.244404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:08.244404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:08.244404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:08.244404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:08.244404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:08.244404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:08.244404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:08.244404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:08.244404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:08.244404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:08.244404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:08.245412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:08.245412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:08.245412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:08.245412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:08.245412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:08.245412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:08.245412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:08.245412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:08.245412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:08.245412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:08.245412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:08.245412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:08.245412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:08.245412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:08.245412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:08.245412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:08.245412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:08.246262 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:08.246262 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:08.246262 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:08.246262 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 204/391] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:08.246262 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4c495a-8063-4e53-9da4-f933a4183ddc None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:08.299103 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:08.302873 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb94179-f575-4311-a686-8f32dda1768a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:08.336483 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcc2ed9-fcc5-4bb9-ae51-dc68672318b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:08.337100 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcc2ed9-fcc5-4bb9-ae51-dc68672318b1 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:08.337376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcc2ed9-fcc5-4bb9-ae51-dc68672318b1 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:08.337647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcc2ed9-fcc5-4bb9-ae51-dc68672318b1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:08.348304 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcc2ed9-fcc5-4bb9-ae51-dc68672318b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:08.349267 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4c495a-8063-4e53-9da4-f933a4183ddc None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:08.349747 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcc2ed9-fcc5-4bb9-ae51-dc68672318b1 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:08.363973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-2bcc2ed9-fcc5-4bb9-ae51-dc68672318b1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:08.363973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:08.363973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:08.363973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:08.363973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:08.363973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:08.363973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:08.363973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:08.363973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:08.363973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:08.363973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:08.363973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:08.363973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:08.363973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:08.363973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:08.363973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:08.365012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:08.365012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:08.365012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:08.365012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:08.365012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:08.365012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:08.365012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:08.365012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:08.365012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:08.365012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:08.365012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:08.365012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:08.365012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:08.365012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:08.365012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:08.365012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:08.365012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:08.366330 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:08.366330 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:08.366330 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:08.366330 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 188/392] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:08.366330 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcc2ed9-fcc5-4bb9-ae51-dc68672318b1 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:08.370737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:08.386496 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b6ef61-6095-4d1f-8527-8300f821f799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:08.387120 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b6ef61-6095-4d1f-8527-8300f821f799 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:08.387397 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b6ef61-6095-4d1f-8527-8300f821f799 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:08.387707 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b6ef61-6095-4d1f-8527-8300f821f799 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:08.412775 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b6ef61-6095-4d1f-8527-8300f821f799 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:08.414209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b6ef61-6095-4d1f-8527-8300f821f799 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:08.428769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-f4b6ef61-6095-4d1f-8527-8300f821f799 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:08.428769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:08.428769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:08.428769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:08.428769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:08.428769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:08.428769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:08.428769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:08.428769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:08.428769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:08.428769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:08.428769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:08.428769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:08.428769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:08.428769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:08.428769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:08.429624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:08.429624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:08.429624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:08.429624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:08.429624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:08.429624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:08.429624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:08.429624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:08.429624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:08.429624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:08.429624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:08.429624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:08.429624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:08.429624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:08.429624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:08.429624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:08.429624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:08.430740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:08.430740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:08.430740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:08.430740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 205/393] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:08.430740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b6ef61-6095-4d1f-8527-8300f821f799 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:08.470153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcc2ed9-fcc5-4bb9-ae51-dc68672318b1 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:08.502713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909bec46-e5af-40c3-b8c1-a07bec2c6f63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:08.503421 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909bec46-e5af-40c3-b8c1-a07bec2c6f63 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:08.503784 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909bec46-e5af-40c3-b8c1-a07bec2c6f63 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:08.504124 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909bec46-e5af-40c3-b8c1-a07bec2c6f63 None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:08.504912 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909bec46-e5af-40c3-b8c1-a07bec2c6f63 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:08.505280 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909bec46-e5af-40c3-b8c1-a07bec2c6f63 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:08.506756 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909bec46-e5af-40c3-b8c1-a07bec2c6f63 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:08.524142 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 189/394] 10.222.0.36 () {56 vars in 1137 bytes} [Mon Oct 31 14:44:08 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:08.525042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909bec46-e5af-40c3-b8c1-a07bec2c6f63 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:08.529819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b6ef61-6095-4d1f-8527-8300f821f799 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:08.533401 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:08.564139 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114b9319-d316-41ac-b45b-7ec0f51a3810 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:08.564763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114b9319-d316-41ac-b45b-7ec0f51a3810 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:08.565037 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114b9319-d316-41ac-b45b-7ec0f51a3810 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:08.565310 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114b9319-d316-41ac-b45b-7ec0f51a3810 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:08.575656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114b9319-d316-41ac-b45b-7ec0f51a3810 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:08.577187 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114b9319-d316-41ac-b45b-7ec0f51a3810 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:08.590257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:08.592304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-114b9319-d316-41ac-b45b-7ec0f51a3810 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:08.592304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:08.592304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:08.592304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:08.592304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:08.592304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:08.592304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:08.592304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:08.592304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:08.592304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:08.592304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:08.592304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:08.592304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:08.592304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:08.592304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:08.592304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:08.593210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:08.593210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:08.593210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:08.593210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:08.593210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:08.593210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:08.593210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:08.593210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:08.593210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:08.593210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:08.593210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:08.593210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:08.593210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:08.593210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:08.593210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:08.593210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:08.593210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:08.594057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:08.594057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:08.594057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:08.594057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 206/395] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:08.594281 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114b9319-d316-41ac-b45b-7ec0f51a3810 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:08.630032 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909bec46-e5af-40c3-b8c1-a07bec2c6f63 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:08.663112 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64ef63e-a3b8-47e9-a8b1-af3e25ede9c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:08.663739 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64ef63e-a3b8-47e9-a8b1-af3e25ede9c4 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:08.664017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64ef63e-a3b8-47e9-a8b1-af3e25ede9c4 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:08.664291 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64ef63e-a3b8-47e9-a8b1-af3e25ede9c4 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:08.664930 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64ef63e-a3b8-47e9-a8b1-af3e25ede9c4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:08.665236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64ef63e-a3b8-47e9-a8b1-af3e25ede9c4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:08.666626 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64ef63e-a3b8-47e9-a8b1-af3e25ede9c4 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:08.677863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 190/396] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:08.678655 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64ef63e-a3b8-47e9-a8b1-af3e25ede9c4 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:08.698236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114b9319-d316-41ac-b45b-7ec0f51a3810 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:08.731131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b240b5f0-7791-4b7f-9d09-d018706510e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:08.732375 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b240b5f0-7791-4b7f-9d09-d018706510e4 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:08.732720 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b240b5f0-7791-4b7f-9d09-d018706510e4 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:08.732989 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b240b5f0-7791-4b7f-9d09-d018706510e4 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:08.733618 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b240b5f0-7791-4b7f-9d09-d018706510e4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:08.733915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b240b5f0-7791-4b7f-9d09-d018706510e4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:08.735259 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b240b5f0-7791-4b7f-9d09-d018706510e4 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:08.746435 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 207/397] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:08.747248 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b240b5f0-7791-4b7f-9d09-d018706510e4 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:08.765897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:08.777161 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64ef63e-a3b8-47e9-a8b1-af3e25ede9c4 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:08.799852 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:08.811969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a98853a-18c1-4c1f-a1ef-3de21bbfed50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:08.812567 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a98853a-18c1-4c1f-a1ef-3de21bbfed50 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:08.812847 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a98853a-18c1-4c1f-a1ef-3de21bbfed50 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:08.813121 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a98853a-18c1-4c1f-a1ef-3de21bbfed50 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:08.813760 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a98853a-18c1-4c1f-a1ef-3de21bbfed50 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:08.814060 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a98853a-18c1-4c1f-a1ef-3de21bbfed50 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:08.815439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a98853a-18c1-4c1f-a1ef-3de21bbfed50 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:08.827444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 191/398] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:08.828277 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a98853a-18c1-4c1f-a1ef-3de21bbfed50 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:08.847580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b240b5f0-7791-4b7f-9d09-d018706510e4 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:08.880943 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56fec64-213e-4a49-955c-5cd2b83002a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:08.881566 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56fec64-213e-4a49-955c-5cd2b83002a7 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:08.881839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56fec64-213e-4a49-955c-5cd2b83002a7 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:08.882108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56fec64-213e-4a49-955c-5cd2b83002a7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:08.892547 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56fec64-213e-4a49-955c-5cd2b83002a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:08.893994 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56fec64-213e-4a49-955c-5cd2b83002a7 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:08.908321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-c56fec64-213e-4a49-955c-5cd2b83002a7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:08.908321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:08.908321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:08.908321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:08.908321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:08.908321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:08.908321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:08.908321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:08.908321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:08.908321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:08.908321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:08.908321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:08.908321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:08.908321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:08.908321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:08.908321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:08.909399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:08.909399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:08.909399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:08.909399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:08.909399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:08.909399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:08.909399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:08.909399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:08.909399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:08.909399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:08.909399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:08.909399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:08.909399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:08.909399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:08.909399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:08.909399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:08.909399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:08.910253 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:08.910253 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:08.910253 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:08.910253 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 208/399] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:08.910253 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56fec64-213e-4a49-955c-5cd2b83002a7 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:08.931857 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a98853a-18c1-4c1f-a1ef-3de21bbfed50 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:08.966578 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efe37a6-29d7-46b8-8626-7e91526ce162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:08.967168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efe37a6-29d7-46b8-8626-7e91526ce162 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:08.967468 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efe37a6-29d7-46b8-8626-7e91526ce162 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:08.967770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efe37a6-29d7-46b8-8626-7e91526ce162 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:08.968410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efe37a6-29d7-46b8-8626-7e91526ce162 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:08.968707 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efe37a6-29d7-46b8-8626-7e91526ce162 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:08.970344 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efe37a6-29d7-46b8-8626-7e91526ce162 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:08.976130 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:08.982137 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 192/400] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:08.982993 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efe37a6-29d7-46b8-8626-7e91526ce162 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:09.013954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56fec64-213e-4a49-955c-5cd2b83002a7 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:09.025534 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:09.048057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0697a74-0999-49a6-9ea6-42cd6d599c4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:09.048679 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0697a74-0999-49a6-9ea6-42cd6d599c4e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:09.048953 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0697a74-0999-49a6-9ea6-42cd6d599c4e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:09.049224 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0697a74-0999-49a6-9ea6-42cd6d599c4e None admin] PATH_INFO: `/v3/projects/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:09.055650 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0697a74-0999-49a6-9ea6-42cd6d599c4e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:09.057080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0697a74-0999-49a6-9ea6-42cd6d599c4e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:09.068344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-e0697a74-0999-49a6-9ea6-42cd6d599c4e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:09.068344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:09.068344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:09.068344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:09.068344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:09.068344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:09.068344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:09.068344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:09.068344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:09.068344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:09.068344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:09.068344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:09.068344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:09.068344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:09.068344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:09.069419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:09.069419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:09.069419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:09.069419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:09.069419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:09.069419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:09.069419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:09.069419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:09.069419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:09.069419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:09.069419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:09.069419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:09.069419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:09.069419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:09.069419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:09.069419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:09.069419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:09.070264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:09.070264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:09.070264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:09.070264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:09.070264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:09.070264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:09.070264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:09.070264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:09.070264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 209/401] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:09.070264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0697a74-0999-49a6-9ea6-42cd6d599c4e None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:09.088266 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efe37a6-29d7-46b8-8626-7e91526ce162 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:09.121853 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7bcbbf-544e-4eb3-94ee-cba2af179a02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:09.122976 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7bcbbf-544e-4eb3-94ee-cba2af179a02 None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:09.123246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7bcbbf-544e-4eb3-94ee-cba2af179a02 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:09.123513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7bcbbf-544e-4eb3-94ee-cba2af179a02 None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:09.124241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7bcbbf-544e-4eb3-94ee-cba2af179a02 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:09.125755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7bcbbf-544e-4eb3-94ee-cba2af179a02 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:09.147145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 193/402] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:44:08 2022] POST /identity/v3/users => generated 247 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:09.148113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7bcbbf-544e-4eb3-94ee-cba2af179a02 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:09.171746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0697a74-0999-49a6-9ea6-42cd6d599c4e None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:09.201195 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:09.210612 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44062275-a1af-41d7-ab62-e6dab7668a86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:09.211412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44062275-a1af-41d7-ab62-e6dab7668a86 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:09.211827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44062275-a1af-41d7-ab62-e6dab7668a86 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:09.212192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44062275-a1af-41d7-ab62-e6dab7668a86 None admin] PATH_INFO: `/v3/projects/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:09.221344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44062275-a1af-41d7-ab62-e6dab7668a86 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:09.223010 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44062275-a1af-41d7-ab62-e6dab7668a86 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:09.236580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-44062275-a1af-41d7-ab62-e6dab7668a86 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:09.236580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:09.236580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:09.236580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:09.236580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:09.236580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:09.236580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:09.236580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:09.236580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:09.236580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:09.236580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:09.236580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:09.236580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:09.236580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:09.236580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:09.237714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:09.237714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:09.237714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:09.237714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:09.237714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:09.237714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:09.237714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:09.237714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:09.237714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:09.237714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:09.237714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:09.237714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:09.237714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:09.237714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:09.237714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:09.237714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:09.237714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:09.243831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:09.243831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:09.243831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:09.243831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:09.243831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:09.243831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:09.243831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:09.243831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:09.243831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 210/403] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:09.243831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44062275-a1af-41d7-ab62-e6dab7668a86 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:09.264093 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:09.268580 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7bcbbf-544e-4eb3-94ee-cba2af179a02 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:09.309125 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a98f115-27fc-407d-9fba-d8f547d2b4ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:09.310267 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a98f115-27fc-407d-9fba-d8f547d2b4ca None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:09.310663 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a98f115-27fc-407d-9fba-d8f547d2b4ca None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:09.310939 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a98f115-27fc-407d-9fba-d8f547d2b4ca None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:09.311741 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a98f115-27fc-407d-9fba-d8f547d2b4ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:09.312047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a98f115-27fc-407d-9fba-d8f547d2b4ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:09.313456 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a98f115-27fc-407d-9fba-d8f547d2b4ca None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:09.326217 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 194/404] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:09.327086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a98f115-27fc-407d-9fba-d8f547d2b4ca None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:09.362190 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44062275-a1af-41d7-ab62-e6dab7668a86 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:09.397610 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98debbbb-be18-4578-a8ed-619bb304eb08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:09.398254 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98debbbb-be18-4578-a8ed-619bb304eb08 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:09.398532 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98debbbb-be18-4578-a8ed-619bb304eb08 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:09.398802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98debbbb-be18-4578-a8ed-619bb304eb08 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:09.407210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98debbbb-be18-4578-a8ed-619bb304eb08 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:09.408680 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98debbbb-be18-4578-a8ed-619bb304eb08 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:09.422394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-98debbbb-be18-4578-a8ed-619bb304eb08 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 31 14:44:09.422394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:09.422394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:09.422394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:09.422394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:09.422394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:09.422394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:09.422394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:09.422394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:09.422394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:09.422394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:09.422394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:09.422394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:09.422394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:09.422394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:09.422394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:09.423472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:09.424493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:09.424493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:09.424493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:09.424493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:09.424493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:09.424493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:09.424493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:09.424493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:09.424493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 31 14:44:09.424493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:09.424493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 211/405] 10.222.0.36 () {56 vars in 1102 bytes} [Mon Oct 31 14:44:09 2022] GET /identity/v3/users/neutron => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:09.425039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98debbbb-be18-4578-a8ed-619bb304eb08 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:09.444677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a98f115-27fc-407d-9fba-d8f547d2b4ca None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:09.479869 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:09.483470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bb7608-eb40-4be7-9426-d36e8e41d7b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:09.484113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bb7608-eb40-4be7-9426-d36e8e41d7b3 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:09.484390 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bb7608-eb40-4be7-9426-d36e8e41d7b3 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:09.484659 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bb7608-eb40-4be7-9426-d36e8e41d7b3 None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:09.486111 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bb7608-eb40-4be7-9426-d36e8e41d7b3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:09.486417 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bb7608-eb40-4be7-9426-d36e8e41d7b3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:09.487865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bb7608-eb40-4be7-9426-d36e8e41d7b3 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:09.511967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 195/406] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:09.511967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bb7608-eb40-4be7-9426-d36e8e41d7b3 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:09.536440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:09.546936 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98debbbb-be18-4578-a8ed-619bb304eb08 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:09.583898 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db88b735-2396-4745-b292-d23c916b7ccf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:09.584500 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db88b735-2396-4745-b292-d23c916b7ccf None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:09.584781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db88b735-2396-4745-b292-d23c916b7ccf None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:09.585058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db88b735-2396-4745-b292-d23c916b7ccf None admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:09.585739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db88b735-2396-4745-b292-d23c916b7ccf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:09.586042 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db88b735-2396-4745-b292-d23c916b7ccf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:09.587474 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db88b735-2396-4745-b292-d23c916b7ccf None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:09.611039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 212/407] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:09.611039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db88b735-2396-4745-b292-d23c916b7ccf None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:09.628059 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bb7608-eb40-4be7-9426-d36e8e41d7b3 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:09.666478 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366c3031-cef2-411e-8fa9-9b75104458bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:09.667365 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366c3031-cef2-411e-8fa9-9b75104458bc None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:09.667365 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366c3031-cef2-411e-8fa9-9b75104458bc None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:09.667732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366c3031-cef2-411e-8fa9-9b75104458bc None admin] PATH_INFO: `/v3/projects/service` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:09.675360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366c3031-cef2-411e-8fa9-9b75104458bc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:09.676847 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366c3031-cef2-411e-8fa9-9b75104458bc None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:09.689627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-366c3031-cef2-411e-8fa9-9b75104458bc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:09.689627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:09.689627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:09.689627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:09.689627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:09.689627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:09.689627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:09.689627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:09.689627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:09.689627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:09.689627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:09.689627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:09.689627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:09.689627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:09.689627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:09.690509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:09.690509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:09.690509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:09.690509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:09.690509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:09.690509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:09.690509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:09.690509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:09.690509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:09.690509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:09.690509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:09.690509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:09.690509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:09.690509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:09.690509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:09.690509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:09.690509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:09.696220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:09.696220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:09.696220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:09.696220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:09.696220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:09.696220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:09.696220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:09.696220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:09.696220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 196/408] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:09.696220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366c3031-cef2-411e-8fa9-9b75104458bc None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:09.724983 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db88b735-2396-4745-b292-d23c916b7ccf None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:09.737971 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:09.766642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60de2d5-91e8-423e-a155-8d686f6c86e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:09.767250 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60de2d5-91e8-423e-a155-8d686f6c86e3 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:09.767530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60de2d5-91e8-423e-a155-8d686f6c86e3 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:09.767825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60de2d5-91e8-423e-a155-8d686f6c86e3 None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:09.768615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60de2d5-91e8-423e-a155-8d686f6c86e3 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:09.768853 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60de2d5-91e8-423e-a155-8d686f6c86e3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:09.770318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60de2d5-91e8-423e-a155-8d686f6c86e3 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:09.782048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:09.793236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 213/409] 10.222.0.36 () {56 vars in 1139 bytes} [Mon Oct 31 14:44:09 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:09.793236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60de2d5-91e8-423e-a155-8d686f6c86e3 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:09.809689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366c3031-cef2-411e-8fa9-9b75104458bc None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:09.844375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f4b731-34b6-42ee-b02a-aeafade4cc99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:09.845377 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f4b731-34b6-42ee-b02a-aeafade4cc99 None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:09.845716 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f4b731-34b6-42ee-b02a-aeafade4cc99 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:09.846050 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f4b731-34b6-42ee-b02a-aeafade4cc99 None admin] PATH_INFO: `/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/c8cace029a92406f850bb0a4ca07c461/roles/88d62c66100a4d04a45e3607135bdd0c` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:09.861352 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f4b731-34b6-42ee-b02a-aeafade4cc99 None admin] RBAC: Authorizing `identity:create_grant(project_id=4118e33b099f4cf0a3e3a46b211ba1d8, user_id=c8cace029a92406f850bb0a4ca07c461, role_id=88d62c66100a4d04a45e3607135bdd0c)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:09.862911 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f4b731-34b6-42ee-b02a-aeafade4cc99 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:09.877773 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 197/410] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:44:09 2022] PUT /identity/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/c8cace029a92406f850bb0a4ca07c461/roles/88d62c66100a4d04a45e3607135bdd0c => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:44:09.884504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f4b731-34b6-42ee-b02a-aeafade4cc99 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:09.902831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60de2d5-91e8-423e-a155-8d686f6c86e3 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:09.944519 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4dfec7-03dc-4c91-8262-d0ae0c67e942 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:09.945476 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4dfec7-03dc-4c91-8262-d0ae0c67e942 None admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:09.945847 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4dfec7-03dc-4c91-8262-d0ae0c67e942 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:09.946218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4dfec7-03dc-4c91-8262-d0ae0c67e942 None admin] PATH_INFO: `/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/993d553a46044c18a4eec6e408c0969a/roles/88d62c66100a4d04a45e3607135bdd0c` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:09.964694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4dfec7-03dc-4c91-8262-d0ae0c67e942 None admin] RBAC: Authorizing `identity:create_grant(project_id=4118e33b099f4cf0a3e3a46b211ba1d8, user_id=993d553a46044c18a4eec6e408c0969a, role_id=88d62c66100a4d04a45e3607135bdd0c)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:09.967314 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4dfec7-03dc-4c91-8262-d0ae0c67e942 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:09.983852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 214/411] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:44:09 2022] PUT /identity/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/993d553a46044c18a4eec6e408c0969a/roles/88d62c66100a4d04a45e3607135bdd0c => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:44:09.985005 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4dfec7-03dc-4c91-8262-d0ae0c67e942 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:10.009500 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:10.036848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f4b731-34b6-42ee-b02a-aeafade4cc99 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:10.094851 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:10.115649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a107df4e-e585-4354-8962-5992e88ebf8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:10.116439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a107df4e-e585-4354-8962-5992e88ebf8e None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:10.116860 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a107df4e-e585-4354-8962-5992e88ebf8e None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:10.117226 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a107df4e-e585-4354-8962-5992e88ebf8e None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:10.118059 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a107df4e-e585-4354-8962-5992e88ebf8e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:10.119440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a107df4e-e585-4354-8962-5992e88ebf8e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:10.121448 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a107df4e-e585-4354-8962-5992e88ebf8e None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:10.150923 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 198/412] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:10.159679 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a107df4e-e585-4354-8962-5992e88ebf8e None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:10.166979 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4dfec7-03dc-4c91-8262-d0ae0c67e942 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:10.216587 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fd552b-547e-4e79-b5fb-359225aef508 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:10.217485 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fd552b-547e-4e79-b5fb-359225aef508 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:10.217970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fd552b-547e-4e79-b5fb-359225aef508 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:10.218486 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fd552b-547e-4e79-b5fb-359225aef508 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:10.230621 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fd552b-547e-4e79-b5fb-359225aef508 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:10.233037 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fd552b-547e-4e79-b5fb-359225aef508 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:10.250293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-64fd552b-547e-4e79-b5fb-359225aef508 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:10.250293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:10.250293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:10.250293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:10.250293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:10.250293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:10.250293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:10.250293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:10.250293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:10.250293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:10.250293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:10.250293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:10.250293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:10.250293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:10.250293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:10.250293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:10.251245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:10.251245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:10.251245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:10.251245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:10.251245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:10.251245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:10.251245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:10.251245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:10.251245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:10.251245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:10.251245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:10.251245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:10.251245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:10.251245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:10.251245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:10.251245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:10.251245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:10.252313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:10.252313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:10.252313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:10.253974 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 215/413] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:10.259861 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fd552b-547e-4e79-b5fb-359225aef508 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:10.283665 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a107df4e-e585-4354-8962-5992e88ebf8e None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:10.323189 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7083a2bd-381b-4602-838a-38cd9fc1e584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:10.324077 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7083a2bd-381b-4602-838a-38cd9fc1e584 None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:10.324431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7083a2bd-381b-4602-838a-38cd9fc1e584 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:10.324831 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7083a2bd-381b-4602-838a-38cd9fc1e584 None admin] PATH_INFO: `/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/a9808dcba0b64027bb70e31d1569957d/roles/88d62c66100a4d04a45e3607135bdd0c` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:10.334406 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:10.343151 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7083a2bd-381b-4602-838a-38cd9fc1e584 None admin] RBAC: Authorizing `identity:create_grant(project_id=4118e33b099f4cf0a3e3a46b211ba1d8, user_id=a9808dcba0b64027bb70e31d1569957d, role_id=88d62c66100a4d04a45e3607135bdd0c)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:10.345185 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7083a2bd-381b-4602-838a-38cd9fc1e584 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:10.362800 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 199/414] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:44:10 2022] PUT /identity/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/a9808dcba0b64027bb70e31d1569957d/roles/88d62c66100a4d04a45e3607135bdd0c => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:44:10.377805 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fd552b-547e-4e79-b5fb-359225aef508 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:10.416861 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29db445-48ba-4c46-9e7f-6c9745c1c710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:10.417489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29db445-48ba-4c46-9e7f-6c9745c1c710 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:10.417768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29db445-48ba-4c46-9e7f-6c9745c1c710 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:10.418040 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29db445-48ba-4c46-9e7f-6c9745c1c710 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:10.418692 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29db445-48ba-4c46-9e7f-6c9745c1c710 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:10.419591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29db445-48ba-4c46-9e7f-6c9745c1c710 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:10.420979 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29db445-48ba-4c46-9e7f-6c9745c1c710 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:10.434685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 216/415] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:10.440637 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7083a2bd-381b-4602-838a-38cd9fc1e584 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:10.502270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:10.588291 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7083a2bd-381b-4602-838a-38cd9fc1e584 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:10.626876 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029eeaaf-72c6-4bdd-b979-45cae41668d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:10.627565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029eeaaf-72c6-4bdd-b979-45cae41668d0 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:10.627862 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029eeaaf-72c6-4bdd-b979-45cae41668d0 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:10.628133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029eeaaf-72c6-4bdd-b979-45cae41668d0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:10.635097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029eeaaf-72c6-4bdd-b979-45cae41668d0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:10.636606 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029eeaaf-72c6-4bdd-b979-45cae41668d0 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:10.648327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-029eeaaf-72c6-4bdd-b979-45cae41668d0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:10.648327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:10.648327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:10.648327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:10.648327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:10.648327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:10.648327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:10.648327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:10.648327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:10.648327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:10.648327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:10.648327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:10.648327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:10.648327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:10.648327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:10.649214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:10.649214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:10.649214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:10.649214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:10.649214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:10.649214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:10.649214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:10.649214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:10.649214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:10.649214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:10.649214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:10.649214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:10.649214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:10.649214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:10.649214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:10.649214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:10.649214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:10.650071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:10.650071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:10.650071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:10.650071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:10.650071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:10.650071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:10.650071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:10.650071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:10.650071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 200/416] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:10.657562 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29db445-48ba-4c46-9e7f-6c9745c1c710 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:10.787634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29db445-48ba-4c46-9e7f-6c9745c1c710 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:10.816583 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:10.826776 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdc698e-6682-4210-812b-94d12f2d95b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:10.827394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdc698e-6682-4210-812b-94d12f2d95b0 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:10.827702 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdc698e-6682-4210-812b-94d12f2d95b0 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:10.827975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdc698e-6682-4210-812b-94d12f2d95b0 None admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:10.828672 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdc698e-6682-4210-812b-94d12f2d95b0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:10.828973 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdc698e-6682-4210-812b-94d12f2d95b0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:10.830420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdc698e-6682-4210-812b-94d12f2d95b0 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:10.846954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 217/417] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:10.852712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029eeaaf-72c6-4bdd-b979-45cae41668d0 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:10.978918 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:10.982905 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029eeaaf-72c6-4bdd-b979-45cae41668d0 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:11.018655 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c15990-ab01-48a0-a234-033553458a8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:11.019322 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c15990-ab01-48a0-a234-033553458a8a None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:11.019626 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c15990-ab01-48a0-a234-033553458a8a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:11.019909 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c15990-ab01-48a0-a234-033553458a8a None admin] PATH_INFO: `/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/14946d97fd384e2a8bff68c4f53c2cfe/roles/88d62c66100a4d04a45e3607135bdd0c` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:11.042244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c15990-ab01-48a0-a234-033553458a8a None admin] RBAC: Authorizing `identity:create_grant(project_id=4118e33b099f4cf0a3e3a46b211ba1d8, user_id=14946d97fd384e2a8bff68c4f53c2cfe, role_id=88d62c66100a4d04a45e3607135bdd0c)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:11.043672 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c15990-ab01-48a0-a234-033553458a8a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:11.068747 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 201/418] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:44:10 2022] PUT /identity/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/14946d97fd384e2a8bff68c4f53c2cfe/roles/88d62c66100a4d04a45e3607135bdd0c => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:44:11.602479 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9a6325-43d2-470a-b8fa-c3253aa0e608 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:11.602982 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9a6325-43d2-470a-b8fa-c3253aa0e608 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:11.603090 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9a6325-43d2-470a-b8fa-c3253aa0e608 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:11.604669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 218/419] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:11.621435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa81760-80b0-4b86-b3a3-ca1777ce0ef6 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:11.621720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa81760-80b0-4b86-b3a3-ca1777ce0ef6 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:11.621999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa81760-80b0-4b86-b3a3-ca1777ce0ef6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:11.649067 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-ffa81760-80b0-4b86-b3a3-ca1777ce0ef6 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:11.650325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ffa81760-80b0-4b86-b3a3-ca1777ce0ef6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:11.744347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 202/420] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:44:12.152370 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9a6325-43d2-470a-b8fa-c3253aa0e608 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:12.273098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9a6325-43d2-470a-b8fa-c3253aa0e608 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:12.310346 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce457170-9cb5-4ff0-ab39-f0fc7a6e4661 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:12.311460 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce457170-9cb5-4ff0-ab39-f0fc7a6e4661 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:12.311967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce457170-9cb5-4ff0-ab39-f0fc7a6e4661 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:12.312436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce457170-9cb5-4ff0-ab39-f0fc7a6e4661 None admin] PATH_INFO: `/v3/roles/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:12.323203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce457170-9cb5-4ff0-ab39-f0fc7a6e4661 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:12.329436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-ce457170-9cb5-4ff0-ab39-f0fc7a6e4661 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:12.329436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:12.329436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:12.329436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:12.329436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:12.329436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:12.329436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:12.329436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:12.329436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:12.329436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:12.329436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:12.329436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:12.329436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:12.329436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:12.329436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:12.329436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:12.330327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:12.330327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:12.330327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:12.330327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:12.330327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:12.330327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:12.330327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:12.330327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:12.330327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:12.330327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:12.330327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:12.330327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:12.330327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:12.330327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:12.330327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:12.330327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:12.331205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:12.331205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:12.331205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:12.331205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:12.331205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:12.331205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:12.331205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:12.331205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:12.331205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:12.331205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:12.331205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:12.331205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:12.333779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 219/421] 10.222.0.36 () {56 vars in 1102 bytes} [Mon Oct 31 14:44:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:12.344954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa81760-80b0-4b86-b3a3-ca1777ce0ef6 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:12.463313 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:12.494290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa81760-80b0-4b86-b3a3-ca1777ce0ef6 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:12.546751 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4024340-a030-4dc6-a819-164f0d468493 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:12.547374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4024340-a030-4dc6-a819-164f0d468493 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:12.547708 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4024340-a030-4dc6-a819-164f0d468493 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:12.547998 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4024340-a030-4dc6-a819-164f0d468493 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:12.548697 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4024340-a030-4dc6-a819-164f0d468493 None admin] RBAC: Adding query filter params (name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:12.549005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4024340-a030-4dc6-a819-164f0d468493 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:12.550428 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4024340-a030-4dc6-a819-164f0d468493 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:12.567101 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 203/422] 10.222.0.36 () {56 vars in 1103 bytes} [Mon Oct 31 14:44:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:12.572954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce457170-9cb5-4ff0-ab39-f0fc7a6e4661 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:12.656358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:12.729872 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce457170-9cb5-4ff0-ab39-f0fc7a6e4661 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:12.746839 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df7ee47-1d94-4371-9438-575855295389 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:12.747824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df7ee47-1d94-4371-9438-575855295389 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:12.748109 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df7ee47-1d94-4371-9438-575855295389 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:12.749592 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 204/423] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:12.760787 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ae167d-ebaf-4d90-9d07-68ac39d18955 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:12.761071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ae167d-ebaf-4d90-9d07-68ac39d18955 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:12.761348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ae167d-ebaf-4d90-9d07-68ac39d18955 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:12.774486 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33b4c2d-849a-461a-a399-e40c05d3161e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:12.775501 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33b4c2d-849a-461a-a399-e40c05d3161e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:12.776021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33b4c2d-849a-461a-a399-e40c05d3161e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:12.776493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33b4c2d-849a-461a-a399-e40c05d3161e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:12.796522 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33b4c2d-849a-461a-a399-e40c05d3161e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:12.798010 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33b4c2d-849a-461a-a399-e40c05d3161e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:12.801861 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-65ae167d-ebaf-4d90-9d07-68ac39d18955 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:12.803214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-65ae167d-ebaf-4d90-9d07-68ac39d18955 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:12.822154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-b33b4c2d-849a-461a-a399-e40c05d3161e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:12.822154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:12.822154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:12.822154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:12.822154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:12.822154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:12.822154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:12.822154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:12.822154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:12.822154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:12.822154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:12.822154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:12.822154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:12.822154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:12.822154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:12.822154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:12.823122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:12.823122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:12.823122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:12.823122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:12.823122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:12.823122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:12.823122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:12.823122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:12.823122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:12.823122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:12.823122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:12.823122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:12.823122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:12.823122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:12.823122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:12.823122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:12.823122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:12.823974 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:12.823974 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:12.823974 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:12.823974 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 220/424] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:12.830004 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33b4c2d-849a-461a-a399-e40c05d3161e None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:12.933734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 205/425] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:44:12.935959 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929faaa0-5448-4622-973f-edb23a5d1171 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:12.936248 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929faaa0-5448-4622-973f-edb23a5d1171 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:12.936528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929faaa0-5448-4622-973f-edb23a5d1171 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:12.938116 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 206/426] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:44:12.943140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b91ddfc-100d-4369-9ba3-834f704000ac None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:12.943424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b91ddfc-100d-4369-9ba3-834f704000ac None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:12.943736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b91ddfc-100d-4369-9ba3-834f704000ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:12.978221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-4b91ddfc-100d-4369-9ba3-834f704000ac None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:12.979444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-4b91ddfc-100d-4369-9ba3-834f704000ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:13.027984 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:13.032497 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33b4c2d-849a-461a-a399-e40c05d3161e None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:13.096629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:13.104466 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 207/427] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:44:13.115109 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e0649c-305b-4531-a5c9-bb4aef833fb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:13.115742 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e0649c-305b-4531-a5c9-bb4aef833fb7 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:13.116018 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e0649c-305b-4531-a5c9-bb4aef833fb7 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:13.116286 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e0649c-305b-4531-a5c9-bb4aef833fb7 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:13.116927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e0649c-305b-4531-a5c9-bb4aef833fb7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:13.117222 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e0649c-305b-4531-a5c9-bb4aef833fb7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:13.118580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e0649c-305b-4531-a5c9-bb4aef833fb7 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:13.132383 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59bf35d-b21b-4764-88e2-037a75f367ba None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:13.132665 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59bf35d-b21b-4764-88e2-037a75f367ba None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:13.132940 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59bf35d-b21b-4764-88e2-037a75f367ba None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:13.134276 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 208/428] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:13.141636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59bf35d-b21b-4764-88e2-037a75f367ba None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:13.143874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 221/429] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:13.146016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c6c827-9ca3-46d4-b103-4cb1c0211f29 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:13.146299 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c6c827-9ca3-46d4-b103-4cb1c0211f29 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:13.146572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c6c827-9ca3-46d4-b103-4cb1c0211f29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:13.178682 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-43c6c827-9ca3-46d4-b103-4cb1c0211f29 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:13.180263 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-43c6c827-9ca3-46d4-b103-4cb1c0211f29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:13.261468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 222/430] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:44:13.262951 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c6c827-9ca3-46d4-b103-4cb1c0211f29 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:13.315056 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59bf35d-b21b-4764-88e2-037a75f367ba None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:13.353996 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf7d112-1c19-454e-b300-fd96778e4b97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:13.355385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf7d112-1c19-454e-b300-fd96778e4b97 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:13.355920 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf7d112-1c19-454e-b300-fd96778e4b97 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:13.357043 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf7d112-1c19-454e-b300-fd96778e4b97 None admin] PATH_INFO: `/v3/users/swift` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:13.365167 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf7d112-1c19-454e-b300-fd96778e4b97 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:13.366871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf7d112-1c19-454e-b300-fd96778e4b97 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:13.380433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-aaf7d112-1c19-454e-b300-fd96778e4b97 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 31 14:44:13.380433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:13.380433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:13.380433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:13.380433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:13.380433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:13.380433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:13.380433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:13.380433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:13.380433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:13.380433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:13.380433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:13.380433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:13.380433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:13.380433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:13.380433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:13.381374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:13.382263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:13.382263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:13.382263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:13.382263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:13.382263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:13.382263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:13.382263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:13.382263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:13.382263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 31 14:44:13.382263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:13.384972 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 209/431] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:44:13 2022] GET /identity/v3/users/swift => generated 83 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:13.391380 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf7d112-1c19-454e-b300-fd96778e4b97 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:13.416136 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:13.428305 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c6c827-9ca3-46d4-b103-4cb1c0211f29 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:13.469438 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:13.476880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4e497e-89d8-42cd-9b72-9aee7a45b5de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:13.477529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4e497e-89d8-42cd-9b72-9aee7a45b5de None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:13.477807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4e497e-89d8-42cd-9b72-9aee7a45b5de None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:13.478077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4e497e-89d8-42cd-9b72-9aee7a45b5de None admin] PATH_INFO: `/v3/roles/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:13.492175 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4e497e-89d8-42cd-9b72-9aee7a45b5de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:13.498967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-fe4e497e-89d8-42cd-9b72-9aee7a45b5de None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:13.498967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:13.498967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:13.498967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:13.498967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:13.498967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:13.498967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:13.498967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:13.498967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:13.498967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:13.498967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:13.498967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:13.498967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:13.498967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:13.498967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:13.498967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:13.499897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:13.499897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:13.499897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:13.499897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:13.499897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:13.499897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:13.499897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:13.499897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:13.499897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:13.499897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:13.499897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:13.499897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:13.499897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:13.499897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:13.499897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:13.499897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:13.500694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:13.500694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:13.500694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:13.500694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:13.500694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:13.500694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:13.500694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:13.500694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:13.500694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:13.500694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:13.500694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:13.500694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:13.500694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 223/432] 10.222.0.36 () {56 vars in 1102 bytes} [Mon Oct 31 14:44:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:13.501386 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4e497e-89d8-42cd-9b72-9aee7a45b5de None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:13.534882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf7d112-1c19-454e-b300-fd96778e4b97 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:13.573164 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fb41de-8f22-49b9-8cdb-80ac59a05e6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:13.574198 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fb41de-8f22-49b9-8cdb-80ac59a05e6a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:13.574681 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fb41de-8f22-49b9-8cdb-80ac59a05e6a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:13.576437 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fb41de-8f22-49b9-8cdb-80ac59a05e6a None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:13.577426 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fb41de-8f22-49b9-8cdb-80ac59a05e6a None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:13.577937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fb41de-8f22-49b9-8cdb-80ac59a05e6a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:13.579698 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fb41de-8f22-49b9-8cdb-80ac59a05e6a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:13.599603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 210/433] 10.222.0.36 () {56 vars in 1135 bytes} [Mon Oct 31 14:44:13 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:13.600601 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fb41de-8f22-49b9-8cdb-80ac59a05e6a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:13.630553 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4e497e-89d8-42cd-9b72-9aee7a45b5de None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:13.669781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f771508-68ac-45c4-a4b8-300d30d7e3dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:13.670529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f771508-68ac-45c4-a4b8-300d30d7e3dd None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:13.670814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f771508-68ac-45c4-a4b8-300d30d7e3dd None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:13.671087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f771508-68ac-45c4-a4b8-300d30d7e3dd None admin] PATH_INFO: `/v3/roles/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:13.682248 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f771508-68ac-45c4-a4b8-300d30d7e3dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:13.688631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-2f771508-68ac-45c4-a4b8-300d30d7e3dd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:13.688631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:13.688631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:13.688631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:13.688631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:13.688631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:13.688631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:13.688631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:13.688631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:13.688631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:13.688631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:13.688631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:13.688631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:13.688631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:13.688631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:13.688631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:13.690294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:13.690294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:13.690294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:13.690294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:13.690294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:13.690294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:13.690294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:13.690294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:13.690294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:13.690294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:13.690294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:13.690294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:13.690294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:13.690294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:13.690294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:13.690294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:13.691878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:13.691878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:13.691878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:13.691878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:13.691878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:13.691878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:13.691878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:13.691878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:13.691878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:13.691878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:13.691878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:13.691878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:13.691878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 224/434] 10.222.0.36 () {56 vars in 1102 bytes} [Mon Oct 31 14:44:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:13.691878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f771508-68ac-45c4-a4b8-300d30d7e3dd None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:13.710074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:13.714037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fb41de-8f22-49b9-8cdb-80ac59a05e6a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:13.737800 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:13.761186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c905ba3-1aed-4349-9a68-2b0b05fa81cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:13.762237 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c905ba3-1aed-4349-9a68-2b0b05fa81cf None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:13.763713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c905ba3-1aed-4349-9a68-2b0b05fa81cf None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:13.764025 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c905ba3-1aed-4349-9a68-2b0b05fa81cf None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:13.764713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c905ba3-1aed-4349-9a68-2b0b05fa81cf None admin] RBAC: Adding query filter params (name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:13.765041 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c905ba3-1aed-4349-9a68-2b0b05fa81cf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:13.766496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c905ba3-1aed-4349-9a68-2b0b05fa81cf None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:13.779804 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 211/435] 10.222.0.36 () {56 vars in 1103 bytes} [Mon Oct 31 14:44:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:13.779804 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c905ba3-1aed-4349-9a68-2b0b05fa81cf None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:13.813542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f771508-68ac-45c4-a4b8-300d30d7e3dd None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:13.854988 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9844239-9285-4349-8f80-83b21610dc9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:13.855546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9844239-9285-4349-8f80-83b21610dc9c None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:13.855851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9844239-9285-4349-8f80-83b21610dc9c None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:13.856121 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9844239-9285-4349-8f80-83b21610dc9c None admin] PATH_INFO: `/v3/roles/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:13.868536 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9844239-9285-4349-8f80-83b21610dc9c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:13.873885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-b9844239-9285-4349-8f80-83b21610dc9c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:13.873885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:13.873885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:13.873885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:13.873885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:13.873885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:13.873885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:13.873885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:13.873885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:13.873885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:13.873885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:13.873885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:13.873885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:13.873885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:13.873885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:13.873885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:13.874794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:13.874794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:13.874794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:13.874794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:13.874794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:13.874794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:13.874794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:13.874794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:13.874794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:13.874794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:13.874794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:13.874794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:13.874794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:13.874794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:13.874794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:13.874794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:13.876047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:13.876047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:13.876047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:13.876047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:13.876047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:13.876047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:13.876047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:13.876047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:13.876047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:13.876047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:13.876047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:13.876047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:13.876047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 225/436] 10.222.0.36 () {56 vars in 1102 bytes} [Mon Oct 31 14:44:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:13.876047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9844239-9285-4349-8f80-83b21610dc9c None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:13.901824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c905ba3-1aed-4349-9a68-2b0b05fa81cf None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:13.936108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf36b19d-75c1-42d8-a200-1a2c04bf59e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:13.936729 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf36b19d-75c1-42d8-a200-1a2c04bf59e5 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:13.937014 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf36b19d-75c1-42d8-a200-1a2c04bf59e5 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:13.937280 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf36b19d-75c1-42d8-a200-1a2c04bf59e5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:13.948646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf36b19d-75c1-42d8-a200-1a2c04bf59e5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:13.950672 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf36b19d-75c1-42d8-a200-1a2c04bf59e5 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:13.959447 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:13.967009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-bf36b19d-75c1-42d8-a200-1a2c04bf59e5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:13.967009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:13.967009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:13.967009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:13.967009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:13.967009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:13.967009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:13.967009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:13.967009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:13.967009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:13.967009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:13.967009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:13.967009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:13.967009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:13.967009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:13.967009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:13.967954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:13.967954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:13.967954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:13.967954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:13.967954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:13.967954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:13.967954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:13.967954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:13.967954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:13.967954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:13.967954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:13.967954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:13.967954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:13.967954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:13.967954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:13.967954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:13.967954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:13.968786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:13.968786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:13.968786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:13.968786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 212/437] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:13.969001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf36b19d-75c1-42d8-a200-1a2c04bf59e5 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:13.995577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9844239-9285-4349-8f80-83b21610dc9c None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:14.001701 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:14.031068 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df71c83-a068-4589-ab3a-67f7c3d4266c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:14.031696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df71c83-a068-4589-ab3a-67f7c3d4266c None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:14.031975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df71c83-a068-4589-ab3a-67f7c3d4266c None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:14.032239 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df71c83-a068-4589-ab3a-67f7c3d4266c None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:14.032893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df71c83-a068-4589-ab3a-67f7c3d4266c None admin] RBAC: Adding query filter params (name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:14.033190 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df71c83-a068-4589-ab3a-67f7c3d4266c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:14.034553 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df71c83-a068-4589-ab3a-67f7c3d4266c None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:14.042246 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 226/438] 10.222.0.36 () {56 vars in 1103 bytes} [Mon Oct 31 14:44:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:14.043039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df71c83-a068-4589-ab3a-67f7c3d4266c None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:14.073663 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf36b19d-75c1-42d8-a200-1a2c04bf59e5 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:14.107888 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b38277-4568-482b-a759-0c643d29844c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:14.108435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b38277-4568-482b-a759-0c643d29844c None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:14.108718 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b38277-4568-482b-a759-0c643d29844c None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:14.108980 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b38277-4568-482b-a759-0c643d29844c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:14.119727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b38277-4568-482b-a759-0c643d29844c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:14.121178 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b38277-4568-482b-a759-0c643d29844c None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:14.135572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-00b38277-4568-482b-a759-0c643d29844c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:14.135572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:14.135572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:14.135572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:14.135572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:14.135572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:14.135572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:14.135572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:14.135572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:14.135572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:14.135572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:14.135572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:14.135572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:14.135572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:14.135572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:14.135572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:14.136549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:14.136549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:14.136549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:14.136549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:14.136549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:14.136549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:14.136549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:14.136549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:14.136549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:14.136549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:14.136549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:14.136549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:14.136549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:14.136549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:14.136549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:14.136549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:14.136549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:14.137375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:14.137375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:14.137375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:14.137375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 213/439] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:14.137815 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095fd9c5-8da3-41a6-b103-6986e2e09ff3 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:14.138256 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095fd9c5-8da3-41a6-b103-6986e2e09ff3 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:14.138434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095fd9c5-8da3-41a6-b103-6986e2e09ff3 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:14.139693 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 214/440] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:14 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:14.140388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095fd9c5-8da3-41a6-b103-6986e2e09ff3 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:14.151437 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df71c83-a068-4589-ab3a-67f7c3d4266c None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:14.181912 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:14.188382 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0e9532-539c-4ae4-b58b-44d7efd4b66f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:14.188992 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0e9532-539c-4ae4-b58b-44d7efd4b66f None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:14.189274 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0e9532-539c-4ae4-b58b-44d7efd4b66f None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:14.189555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0e9532-539c-4ae4-b58b-44d7efd4b66f None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:14.190214 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0e9532-539c-4ae4-b58b-44d7efd4b66f None admin] RBAC: Adding query filter params (name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:14.190515 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0e9532-539c-4ae4-b58b-44d7efd4b66f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:14.191918 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0e9532-539c-4ae4-b58b-44d7efd4b66f None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:14.199789 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 227/441] 10.222.0.36 () {56 vars in 1103 bytes} [Mon Oct 31 14:44:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:14.201638 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0e9532-539c-4ae4-b58b-44d7efd4b66f None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:14.227792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:14.245005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095fd9c5-8da3-41a6-b103-6986e2e09ff3 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:14.278937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3595e6c-484a-4122-8954-d7bca01b6bd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:14.279518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3595e6c-484a-4122-8954-d7bca01b6bd1 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:14.279826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3595e6c-484a-4122-8954-d7bca01b6bd1 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:14.280089 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3595e6c-484a-4122-8954-d7bca01b6bd1 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:14.280739 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3595e6c-484a-4122-8954-d7bca01b6bd1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:14.281036 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3595e6c-484a-4122-8954-d7bca01b6bd1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:14.282437 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3595e6c-484a-4122-8954-d7bca01b6bd1 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:14.293564 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 215/442] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:14.294879 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3595e6c-484a-4122-8954-d7bca01b6bd1 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:14.307230 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0e9532-539c-4ae4-b58b-44d7efd4b66f None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:14.341756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4935071-f862-4751-94bb-dc66099de876 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:14.342374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4935071-f862-4751-94bb-dc66099de876 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:14.342657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4935071-f862-4751-94bb-dc66099de876 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:14.342930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4935071-f862-4751-94bb-dc66099de876 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:14.353481 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4935071-f862-4751-94bb-dc66099de876 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:14.354917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4935071-f862-4751-94bb-dc66099de876 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:14.369756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-f4935071-f862-4751-94bb-dc66099de876 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:14.369756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:14.369756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:14.369756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:14.369756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:14.369756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:14.369756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:14.369756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:14.369756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:14.369756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:14.369756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:14.369756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:14.369756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:14.369756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:14.369756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:14.369756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:14.370832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:14.370832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:14.370832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:14.370832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:14.370832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:14.370832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:14.370832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:14.370832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:14.370832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:14.370832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:14.370832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:14.370832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:14.370832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:14.370832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:14.370832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:14.370832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:14.370832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:14.371739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:14.371739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:14.371739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:14.371739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 228/443] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:14.372492 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52599474-9c02-46ce-bebc-3a0fdce4865c None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:14.372773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52599474-9c02-46ce-bebc-3a0fdce4865c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:14.373047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52599474-9c02-46ce-bebc-3a0fdce4865c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:14.390405 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-52599474-9c02-46ce-bebc-3a0fdce4865c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:14.391631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-52599474-9c02-46ce-bebc-3a0fdce4865c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:14.396438 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3595e6c-484a-4122-8954-d7bca01b6bd1 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:14.430981 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d7340a-eed5-4011-9e45-a9bec12fa474 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:14.431594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d7340a-eed5-4011-9e45-a9bec12fa474 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:14.431878 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d7340a-eed5-4011-9e45-a9bec12fa474 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:14.432151 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d7340a-eed5-4011-9e45-a9bec12fa474 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:14.432787 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d7340a-eed5-4011-9e45-a9bec12fa474 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:14.433088 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d7340a-eed5-4011-9e45-a9bec12fa474 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:14.434513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d7340a-eed5-4011-9e45-a9bec12fa474 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:14.439062 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:14.446213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 216/444] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:14.447016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d7340a-eed5-4011-9e45-a9bec12fa474 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:14.452069 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:14.466975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 229/445] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:44:14.467950 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52599474-9c02-46ce-bebc-3a0fdce4865c None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:14.572656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52599474-9c02-46ce-bebc-3a0fdce4865c None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:14.574173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d7340a-eed5-4011-9e45-a9bec12fa474 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:14.608794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5e6530-0036-4183-8b57-ab8cc21d5228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:14.609410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5e6530-0036-4183-8b57-ab8cc21d5228 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:14.609691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5e6530-0036-4183-8b57-ab8cc21d5228 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:14.609961 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5e6530-0036-4183-8b57-ab8cc21d5228 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:14.610359 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a82537-efe5-40ab-9f1f-bb261a06c905 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:14.611052 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a82537-efe5-40ab-9f1f-bb261a06c905 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:14.611393 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a82537-efe5-40ab-9f1f-bb261a06c905 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:14.611755 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a82537-efe5-40ab-9f1f-bb261a06c905 None admin] PATH_INFO: `/v3/projects/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:14.618669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a82537-efe5-40ab-9f1f-bb261a06c905 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:14.620176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5e6530-0036-4183-8b57-ab8cc21d5228 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:14.620288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a82537-efe5-40ab-9f1f-bb261a06c905 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:14.621720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5e6530-0036-4183-8b57-ab8cc21d5228 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:14.632513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-78a82537-efe5-40ab-9f1f-bb261a06c905 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:14.632513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:14.632513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:14.632513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:14.632513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:14.632513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:14.632513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:14.632513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:14.632513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:14.632513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:14.632513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:14.632513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:14.632513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:14.632513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:14.632513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:14.633530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:14.633530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:14.633530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:14.633530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:14.633530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:14.633530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:14.633530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:14.633530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:14.633530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:14.633530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:14.633530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:14.633530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:14.633530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:14.633530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:14.633530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:14.633530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:14.633530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:14.634579 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:14.634579 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:14.634579 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:14.634579 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:14.634579 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:14.634579 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:14.634579 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:14.634579 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:14.641251 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 230/446] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:14.641251 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a82537-efe5-40ab-9f1f-bb261a06c905 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:14.641460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-da5e6530-0036-4183-8b57-ab8cc21d5228 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:14.641460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:14.641460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:14.641460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:14.641460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:14.641460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:14.641460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:14.641460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:14.641460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:14.641460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:14.641460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:14.641460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:14.641460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:14.641460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:14.641460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:14.641460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:14.642260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:14.642260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:14.642260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:14.642260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:14.642260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:14.642260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:14.642260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:14.642260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:14.642260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:14.642260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:14.642260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:14.642260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:14.642260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:14.642260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:14.642260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:14.642260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:14.642260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:14.643107 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:14.643107 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:14.643107 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:14.643107 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 217/447] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:14.643107 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5e6530-0036-4183-8b57-ab8cc21d5228 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:14.679939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:14.704459 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:14.748060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a82537-efe5-40ab-9f1f-bb261a06c905 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:14.751192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5e6530-0036-4183-8b57-ab8cc21d5228 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:14.781432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6095f046-f069-4f54-a95e-ad3effe27af5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:14.782034 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6095f046-f069-4f54-a95e-ad3effe27af5 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:14.782311 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6095f046-f069-4f54-a95e-ad3effe27af5 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:14.782584 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6095f046-f069-4f54-a95e-ad3effe27af5 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:14.783211 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6095f046-f069-4f54-a95e-ad3effe27af5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:14.783510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6095f046-f069-4f54-a95e-ad3effe27af5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:14.784832 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7a2734-01e0-411d-bf4b-d90df116f221 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:14.784969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6095f046-f069-4f54-a95e-ad3effe27af5 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:14.785439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7a2734-01e0-411d-bf4b-d90df116f221 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:14.785712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7a2734-01e0-411d-bf4b-d90df116f221 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:14.785993 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7a2734-01e0-411d-bf4b-d90df116f221 None admin] PATH_INFO: `/v3/users/nova` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:14.792767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7a2734-01e0-411d-bf4b-d90df116f221 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:14.794819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7a2734-01e0-411d-bf4b-d90df116f221 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:14.796705 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 231/448] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:14.797488 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6095f046-f069-4f54-a95e-ad3effe27af5 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:14.811113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-5a7a2734-01e0-411d-bf4b-d90df116f221 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 31 14:44:14.811113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:14.811113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:14.811113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:14.811113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:14.811113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:14.811113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:14.811113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:14.811113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:14.811113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:14.811113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:14.811113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:14.811113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:14.811113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:14.811113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:14.811113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:14.812119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:14.813001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:14.813001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:14.813001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:14.813001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:14.813001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:14.813001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:14.813001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:14.813001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:14.813001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 31 14:44:14.813001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:14.813001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 218/449] 10.222.0.36 () {56 vars in 1093 bytes} [Mon Oct 31 14:44:14 2022] GET /identity/v3/users/nova => generated 82 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:14.813001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7a2734-01e0-411d-bf4b-d90df116f221 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:14.894453 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:14.900830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6095f046-f069-4f54-a95e-ad3effe27af5 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:14.912690 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7a2734-01e0-411d-bf4b-d90df116f221 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:14.928756 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:14.936554 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8306a068-b746-4c63-83f5-46a08b400025 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:14.937154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8306a068-b746-4c63-83f5-46a08b400025 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:14.937430 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8306a068-b746-4c63-83f5-46a08b400025 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:14.937695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8306a068-b746-4c63-83f5-46a08b400025 None admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:14.938381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8306a068-b746-4c63-83f5-46a08b400025 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:14.938678 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8306a068-b746-4c63-83f5-46a08b400025 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:14.940135 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8306a068-b746-4c63-83f5-46a08b400025 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:14.946323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d7682c-81bf-4e4f-960d-56fb4f386ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:14.946916 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d7682c-81bf-4e4f-960d-56fb4f386ce7 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:14.947193 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d7682c-81bf-4e4f-960d-56fb4f386ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:14.947462 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d7682c-81bf-4e4f-960d-56fb4f386ce7 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:14.948197 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d7682c-81bf-4e4f-960d-56fb4f386ce7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:14.948496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d7682c-81bf-4e4f-960d-56fb4f386ce7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:14.949841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d7682c-81bf-4e4f-960d-56fb4f386ce7 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:14.952183 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 232/450] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:14.952978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8306a068-b746-4c63-83f5-46a08b400025 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:14.966875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 219/451] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:14.967722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d7682c-81bf-4e4f-960d-56fb4f386ce7 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:15.050862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8306a068-b746-4c63-83f5-46a08b400025 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:15.064617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d7682c-81bf-4e4f-960d-56fb4f386ce7 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:15.085149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b916b2a0-9302-470e-b069-2ce7a7f304b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:15.085772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b916b2a0-9302-470e-b069-2ce7a7f304b1 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:15.086055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b916b2a0-9302-470e-b069-2ce7a7f304b1 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:15.086334 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b916b2a0-9302-470e-b069-2ce7a7f304b1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:15.097481 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b916b2a0-9302-470e-b069-2ce7a7f304b1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:15.098437 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb2c3d1-8f23-4335-bcb3-ad4b0136668a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:15.099056 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb2c3d1-8f23-4335-bcb3-ad4b0136668a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:15.099336 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb2c3d1-8f23-4335-bcb3-ad4b0136668a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:15.099641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb2c3d1-8f23-4335-bcb3-ad4b0136668a None admin] PATH_INFO: `/v3/users/cinder` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:15.102327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-b916b2a0-9302-470e-b069-2ce7a7f304b1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:15.102327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:15.102327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:15.102327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:15.102327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:15.102327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:15.102327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:15.102327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:15.102327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:15.102327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:15.102327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:15.102327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:15.102327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:15.102327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:15.102327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:15.102327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:15.103343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:15.103343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:15.103343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:15.103343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:15.103343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:15.103343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:15.103343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:15.103343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:15.103343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:15.103343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:15.103343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:15.103343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:15.103343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:15.103343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:15.103343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:15.103343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:15.104165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:15.104165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:15.104165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:15.104165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:15.104165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:15.104165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:15.104165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:15.104165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:15.104165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:15.104165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:15.104165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:15.104165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:15.104165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 233/452] 10.222.0.36 () {56 vars in 1102 bytes} [Mon Oct 31 14:44:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:15.104804 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b916b2a0-9302-470e-b069-2ce7a7f304b1 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:15.106809 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb2c3d1-8f23-4335-bcb3-ad4b0136668a None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:15.108287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb2c3d1-8f23-4335-bcb3-ad4b0136668a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:15.119836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-4eb2c3d1-8f23-4335-bcb3-ad4b0136668a None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 31 14:44:15.119836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:15.119836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:15.119836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:15.119836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:15.119836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:15.119836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:15.119836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:15.119836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:15.119836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:15.119836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:15.119836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:15.119836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:15.119836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:15.119836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:15.119836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:15.120932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:15.122543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:15.122543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:15.122543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:15.122543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:15.122543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:15.122543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:15.122543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:15.122543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:15.122543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 31 14:44:15.122543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:15.122543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 220/453] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:44:14 2022] GET /identity/v3/users/cinder => generated 84 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:15.122543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb2c3d1-8f23-4335-bcb3-ad4b0136668a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:15.127328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:15.156910 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:15.212183 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b916b2a0-9302-470e-b069-2ce7a7f304b1 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:15.221912 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb2c3d1-8f23-4335-bcb3-ad4b0136668a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:15.245226 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46fcd653-1997-4000-b00b-1539da17c975 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:15.245833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fcd653-1997-4000-b00b-1539da17c975 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:15.246111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fcd653-1997-4000-b00b-1539da17c975 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:15.246387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fcd653-1997-4000-b00b-1539da17c975 None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:15.247710 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46fcd653-1997-4000-b00b-1539da17c975 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:15.248016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46fcd653-1997-4000-b00b-1539da17c975 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:15.249428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46fcd653-1997-4000-b00b-1539da17c975 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:15.255475 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30673644-7bff-4936-b8d8-d4e8e6e9a0c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:15.256143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30673644-7bff-4936-b8d8-d4e8e6e9a0c3 None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:15.256416 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30673644-7bff-4936-b8d8-d4e8e6e9a0c3 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:15.256685 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30673644-7bff-4936-b8d8-d4e8e6e9a0c3 None admin] PATH_INFO: `/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/f4ae5e20cc5b40ccbdb951f908f5080d/roles/88d62c66100a4d04a45e3607135bdd0c` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:15.266942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 234/454] 10.222.0.36 () {56 vars in 1133 bytes} [Mon Oct 31 14:44:15 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:15.267777 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46fcd653-1997-4000-b00b-1539da17c975 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:15.271760 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30673644-7bff-4936-b8d8-d4e8e6e9a0c3 None admin] RBAC: Authorizing `identity:create_grant(project_id=4118e33b099f4cf0a3e3a46b211ba1d8, user_id=f4ae5e20cc5b40ccbdb951f908f5080d, role_id=88d62c66100a4d04a45e3607135bdd0c)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:15.273201 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30673644-7bff-4936-b8d8-d4e8e6e9a0c3 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:15.285886 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 221/455] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:44:15 2022] PUT /identity/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/f4ae5e20cc5b40ccbdb951f908f5080d/roles/88d62c66100a4d04a45e3607135bdd0c => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:44:15.286681 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30673644-7bff-4936-b8d8-d4e8e6e9a0c3 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:15.376841 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:15.386259 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:15.407410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30673644-7bff-4936-b8d8-d4e8e6e9a0c3 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:15.408056 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46fcd653-1997-4000-b00b-1539da17c975 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:15.446386 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d23c2a-d755-4b5d-b8ac-474db4ded034 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:15.447001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d23c2a-d755-4b5d-b8ac-474db4ded034 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:15.447281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d23c2a-d755-4b5d-b8ac-474db4ded034 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:15.447584 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d23c2a-d755-4b5d-b8ac-474db4ded034 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:15.447680 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ce183f-235d-482f-988d-63699cc1d60a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:15.448261 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d23c2a-d755-4b5d-b8ac-474db4ded034 None admin] RBAC: Adding query filter params (name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:15.448366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ce183f-235d-482f-988d-63699cc1d60a None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:15.448553 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ce183f-235d-482f-988d-63699cc1d60a None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:15.448643 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d23c2a-d755-4b5d-b8ac-474db4ded034 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:15.448823 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ce183f-235d-482f-988d-63699cc1d60a None admin] PATH_INFO: `/v3/users/glance` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:15.450065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d23c2a-d755-4b5d-b8ac-474db4ded034 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:15.459939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ce183f-235d-482f-988d-63699cc1d60a None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:15.464765 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ce183f-235d-482f-988d-63699cc1d60a None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:15.464862 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 222/456] 10.222.0.36 () {56 vars in 1103 bytes} [Mon Oct 31 14:44:15 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:15.464862 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d23c2a-d755-4b5d-b8ac-474db4ded034 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:15.474848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-c0ce183f-235d-482f-988d-63699cc1d60a None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 31 14:44:15.474848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:15.474848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:15.474848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:15.474848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:15.474848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:15.474848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:15.474848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:15.474848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:15.474848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:15.474848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:15.474848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:15.474848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:15.474848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:15.474848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:15.474848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:15.475826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:15.476781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:15.476781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:15.476781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:15.476781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:15.476781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:15.476781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:15.476781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:15.476781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:15.476781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 31 14:44:15.476781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:15.476781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 235/457] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:44:15 2022] GET /identity/v3/users/glance => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:15.477623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ce183f-235d-482f-988d-63699cc1d60a None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:15.577277 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d23c2a-d755-4b5d-b8ac-474db4ded034 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:15.589985 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ce183f-235d-482f-988d-63699cc1d60a None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:15.617034 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b788209f-d6ec-4047-b6b0-1c38fe44b068 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:15.617644 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b788209f-d6ec-4047-b6b0-1c38fe44b068 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:15.617926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b788209f-d6ec-4047-b6b0-1c38fe44b068 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:15.618434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b788209f-d6ec-4047-b6b0-1c38fe44b068 None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:15.618929 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b788209f-d6ec-4047-b6b0-1c38fe44b068 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:15.619227 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b788209f-d6ec-4047-b6b0-1c38fe44b068 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:15.620749 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b788209f-d6ec-4047-b6b0-1c38fe44b068 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:15.627452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47209073-83d8-499b-914e-e51cee84e2b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:15.628087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47209073-83d8-499b-914e-e51cee84e2b6 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:15.628369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47209073-83d8-499b-914e-e51cee84e2b6 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:15.628646 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47209073-83d8-499b-914e-e51cee84e2b6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:15.631341 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:15.643713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 223/458] 10.222.0.36 () {56 vars in 1137 bytes} [Mon Oct 31 14:44:15 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:15.644582 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b788209f-d6ec-4047-b6b0-1c38fe44b068 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:15.654396 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:15.708483 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47209073-83d8-499b-914e-e51cee84e2b6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:15.710476 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47209073-83d8-499b-914e-e51cee84e2b6 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:15.768176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b788209f-d6ec-4047-b6b0-1c38fe44b068 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:15.804042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1c97f8-4a62-4e75-885a-ef546ef1eda9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:15.804665 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1c97f8-4a62-4e75-885a-ef546ef1eda9 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:15.804946 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1c97f8-4a62-4e75-885a-ef546ef1eda9 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:15.805218 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1c97f8-4a62-4e75-885a-ef546ef1eda9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:15.814589 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 236/459] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:44:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 31 14:44:15.816588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1c97f8-4a62-4e75-885a-ef546ef1eda9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:15.818037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1c97f8-4a62-4e75-885a-ef546ef1eda9 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:15.819897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47209073-83d8-499b-914e-e51cee84e2b6 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:15.833064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-ac1c97f8-4a62-4e75-885a-ef546ef1eda9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:15.833064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:15.833064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:15.833064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:15.833064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:15.833064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:15.833064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:15.833064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:15.833064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:15.833064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:15.833064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:15.833064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:15.833064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:15.833064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:15.833064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:15.833064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:15.833974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:15.833974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:15.833974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:15.833974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:15.833974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:15.833974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:15.833974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:15.833974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:15.833974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:15.833974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:15.833974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:15.833974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:15.833974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:15.833974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:15.833974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:15.833974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:15.833974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:15.835502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:15.835502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:15.835502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:15.835502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 224/460] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:15.835502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1c97f8-4a62-4e75-885a-ef546ef1eda9 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:15.862476 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:15.896625 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:15.936901 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47209073-83d8-499b-914e-e51cee84e2b6 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:15.939400 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1c97f8-4a62-4e75-885a-ef546ef1eda9 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:15.973873 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c14cf4-827e-4e9f-8cce-9b59779362fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:15.974406 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c14cf4-827e-4e9f-8cce-9b59779362fc None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:15.974678 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c14cf4-827e-4e9f-8cce-9b59779362fc None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:15.974942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c14cf4-827e-4e9f-8cce-9b59779362fc None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:15.975723 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c14cf4-827e-4e9f-8cce-9b59779362fc None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:15.976037 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c14cf4-827e-4e9f-8cce-9b59779362fc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:15.976319 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42245c75-e138-4495-ae87-050ffa7cf8c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:15.976919 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42245c75-e138-4495-ae87-050ffa7cf8c2 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:15.977199 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42245c75-e138-4495-ae87-050ffa7cf8c2 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:15.977469 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c14cf4-827e-4e9f-8cce-9b59779362fc None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:15.977573 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42245c75-e138-4495-ae87-050ffa7cf8c2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:16.001628 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 237/461] 10.222.0.36 () {56 vars in 1137 bytes} [Mon Oct 31 14:44:15 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:16.002352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c14cf4-827e-4e9f-8cce-9b59779362fc None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:16.051134 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42245c75-e138-4495-ae87-050ffa7cf8c2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:16.052878 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42245c75-e138-4495-ae87-050ffa7cf8c2 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:16.100064 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:16.110592 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c14cf4-827e-4e9f-8cce-9b59779362fc None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:16.114473 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:16.147236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b710dcd-485b-4e60-a745-fbf1dd720acc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:16.147887 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b710dcd-485b-4e60-a745-fbf1dd720acc None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:16.148170 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b710dcd-485b-4e60-a745-fbf1dd720acc None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:16.148436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b710dcd-485b-4e60-a745-fbf1dd720acc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:16.159520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b710dcd-485b-4e60-a745-fbf1dd720acc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:16.160975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b710dcd-485b-4e60-a745-fbf1dd720acc None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:16.161457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 225/462] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:44:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 31 14:44:16.162296 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42245c75-e138-4495-ae87-050ffa7cf8c2 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:16.183278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-6b710dcd-485b-4e60-a745-fbf1dd720acc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:16.183278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:16.183278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:16.183278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:16.183278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:16.183278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:16.183278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:16.183278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:16.183278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:16.183278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:16.183278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:16.183278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:16.183278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:16.183278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:16.183278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:16.183278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:16.184829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:16.184829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:16.184829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:16.184829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:16.184829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:16.184829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:16.184829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:16.184829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:16.184829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:16.184829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:16.184829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:16.184829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:16.184829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:16.184829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:16.184829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:16.184829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:16.184829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:16.190532 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:16.190532 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:16.190532 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:16.190532 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 238/463] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:16.190532 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b710dcd-485b-4e60-a745-fbf1dd720acc None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:16.275386 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42245c75-e138-4495-ae87-050ffa7cf8c2 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:16.296916 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b710dcd-485b-4e60-a745-fbf1dd720acc None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:16.312690 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac113b9-df3f-4c8f-b17d-93a95f30d9eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:16.313292 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac113b9-df3f-4c8f-b17d-93a95f30d9eb None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:16.313665 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac113b9-df3f-4c8f-b17d-93a95f30d9eb None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:16.313860 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac113b9-df3f-4c8f-b17d-93a95f30d9eb None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:16.314475 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac113b9-df3f-4c8f-b17d-93a95f30d9eb None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:16.314784 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac113b9-df3f-4c8f-b17d-93a95f30d9eb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:16.316188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac113b9-df3f-4c8f-b17d-93a95f30d9eb None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:16.328206 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 226/464] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:16.329024 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac113b9-df3f-4c8f-b17d-93a95f30d9eb None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:16.338808 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542cec79-9b13-4469-a525-69e0a6a80b9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:16.339423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542cec79-9b13-4469-a525-69e0a6a80b9f None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:16.340026 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542cec79-9b13-4469-a525-69e0a6a80b9f None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:16.340026 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542cec79-9b13-4469-a525-69e0a6a80b9f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:16.349062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:16.357595 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:16.417744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542cec79-9b13-4469-a525-69e0a6a80b9f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:16.420121 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542cec79-9b13-4469-a525-69e0a6a80b9f None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:16.435600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac113b9-df3f-4c8f-b17d-93a95f30d9eb None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:16.470964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7642fe-13bb-4e4e-81ac-b9079c8678bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:16.471613 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7642fe-13bb-4e4e-81ac-b9079c8678bf None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:16.471898 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7642fe-13bb-4e4e-81ac-b9079c8678bf None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:16.472171 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7642fe-13bb-4e4e-81ac-b9079c8678bf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:16.483741 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7642fe-13bb-4e4e-81ac-b9079c8678bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:16.485187 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7642fe-13bb-4e4e-81ac-b9079c8678bf None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:16.499765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-ff7642fe-13bb-4e4e-81ac-b9079c8678bf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:16.499765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:16.499765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:16.499765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:16.499765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:16.499765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:16.499765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:16.499765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:16.499765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:16.499765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:16.499765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:16.499765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:16.499765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:16.499765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:16.499765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:16.499765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:16.500658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:16.500658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:16.500658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:16.500658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:16.500658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:16.500658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:16.500658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:16.500658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:16.500658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:16.500658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:16.500658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:16.500658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:16.500658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:16.500658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:16.500658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:16.500658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:16.500658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:16.501524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:16.501524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:16.501524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:16.501524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 227/465] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:16.502181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7642fe-13bb-4e4e-81ac-b9079c8678bf None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:16.542953 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 239/466] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:44:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 31 14:44:16.542953 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542cec79-9b13-4469-a525-69e0a6a80b9f None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:16.582056 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:16.589446 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:16.616766 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7642fe-13bb-4e4e-81ac-b9079c8678bf None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:16.648472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542cec79-9b13-4469-a525-69e0a6a80b9f None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:16.651720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a89a1da-7117-4a99-b713-5394244e7d7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:16.652326 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a89a1da-7117-4a99-b713-5394244e7d7c None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:16.652606 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a89a1da-7117-4a99-b713-5394244e7d7c None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:16.652879 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a89a1da-7117-4a99-b713-5394244e7d7c None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:16.653591 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a89a1da-7117-4a99-b713-5394244e7d7c None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:16.653862 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a89a1da-7117-4a99-b713-5394244e7d7c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:16.655232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a89a1da-7117-4a99-b713-5394244e7d7c None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:16.667783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 228/467] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:16.673119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a89a1da-7117-4a99-b713-5394244e7d7c None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:16.684999 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b12cec8-dc5a-4113-9194-b19e78f632a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:16.685680 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b12cec8-dc5a-4113-9194-b19e78f632a8 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:16.686188 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b12cec8-dc5a-4113-9194-b19e78f632a8 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:16.686294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b12cec8-dc5a-4113-9194-b19e78f632a8 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:16.693580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b12cec8-dc5a-4113-9194-b19e78f632a8 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:16.695007 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b12cec8-dc5a-4113-9194-b19e78f632a8 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:16.706770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-8b12cec8-dc5a-4113-9194-b19e78f632a8 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 31 14:44:16.706770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:16.706770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:16.706770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:16.706770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:16.706770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:16.706770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:16.706770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:16.706770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:16.706770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:16.706770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:16.706770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:16.706770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:16.706770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:16.706770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:16.706770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:16.707695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:16.709053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:16.709053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:16.709053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:16.709053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:16.709053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:16.709053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:16.709053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:16.709053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:16.709053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 31 14:44:16.709053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:16.712351 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 240/468] 10.222.0.36 () {56 vars in 1102 bytes} [Mon Oct 31 14:44:16 2022] GET /identity/v3/users/neutron => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:16.713168 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b12cec8-dc5a-4113-9194-b19e78f632a8 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:16.784370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a89a1da-7117-4a99-b713-5394244e7d7c None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:16.814449 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:16.820901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f191a81-2194-44f9-9b1e-9da7332b409f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:16.821515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f191a81-2194-44f9-9b1e-9da7332b409f None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:16.821799 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f191a81-2194-44f9-9b1e-9da7332b409f None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:16.822075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f191a81-2194-44f9-9b1e-9da7332b409f None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:16.822627 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b12cec8-dc5a-4113-9194-b19e78f632a8 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:16.822735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f191a81-2194-44f9-9b1e-9da7332b409f None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:16.823011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f191a81-2194-44f9-9b1e-9da7332b409f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:16.824435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f191a81-2194-44f9-9b1e-9da7332b409f None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:16.826121 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:16.837364 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 229/469] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:16.838184 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f191a81-2194-44f9-9b1e-9da7332b409f None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:16.871468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab3bda4-90ea-49fb-b0fd-d6369c7a8bef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:16.872703 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab3bda4-90ea-49fb-b0fd-d6369c7a8bef None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:16.872979 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab3bda4-90ea-49fb-b0fd-d6369c7a8bef None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:16.873247 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab3bda4-90ea-49fb-b0fd-d6369c7a8bef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:16.883843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab3bda4-90ea-49fb-b0fd-d6369c7a8bef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:16.885274 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab3bda4-90ea-49fb-b0fd-d6369c7a8bef None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:16.902078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-6ab3bda4-90ea-49fb-b0fd-d6369c7a8bef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:16.902078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:16.902078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:16.902078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:16.902078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:16.902078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:16.902078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:16.902078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:16.902078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:16.902078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:16.902078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:16.902078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:16.902078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:16.902078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:16.902078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:16.902078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:16.903151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:16.903151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:16.903151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:16.903151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:16.903151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:16.903151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:16.903151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:16.903151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:16.903151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:16.903151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:16.903151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:16.903151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:16.903151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:16.903151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:16.903151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:16.903151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:16.903151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:16.904228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:16.904228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:16.904228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:16.905574 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 241/470] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:16.906490 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab3bda4-90ea-49fb-b0fd-d6369c7a8bef None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:16.952564 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f191a81-2194-44f9-9b1e-9da7332b409f None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:16.987836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e820db65-1612-4765-ac25-c6a89b14b5aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:16.988464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e820db65-1612-4765-ac25-c6a89b14b5aa None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:16.988744 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e820db65-1612-4765-ac25-c6a89b14b5aa None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:16.989349 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e820db65-1612-4765-ac25-c6a89b14b5aa None admin] PATH_INFO: `/v3/projects/service` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:16.996011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e820db65-1612-4765-ac25-c6a89b14b5aa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:16.998033 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e820db65-1612-4765-ac25-c6a89b14b5aa None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:17.010235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-e820db65-1612-4765-ac25-c6a89b14b5aa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:17.010235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:17.010235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:17.010235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:17.010235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:17.010235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:17.010235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:17.010235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:17.010235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:17.010235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:17.010235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:17.010235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:17.010235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:17.010235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:17.010235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:17.011287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:17.011287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:17.011287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:17.011287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:17.011287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:17.011287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:17.011287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:17.011287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:17.011287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:17.011287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:17.011287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:17.011287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:17.011287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:17.011287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:17.011287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:17.011287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:17.011287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:17.012414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:17.012414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:17.012414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:17.012414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:17.012414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:17.012414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:17.012414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:17.012414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:17.018269 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 230/471] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:17.018269 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e820db65-1612-4765-ac25-c6a89b14b5aa None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:17.023417 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab3bda4-90ea-49fb-b0fd-d6369c7a8bef None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:17.060537 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042efec8-9fa3-4483-afc5-59d365e88913 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:17.061056 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042efec8-9fa3-4483-afc5-59d365e88913 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:17.061328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042efec8-9fa3-4483-afc5-59d365e88913 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:17.061595 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042efec8-9fa3-4483-afc5-59d365e88913 None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:17.062315 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042efec8-9fa3-4483-afc5-59d365e88913 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:17.062614 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042efec8-9fa3-4483-afc5-59d365e88913 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:17.064257 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042efec8-9fa3-4483-afc5-59d365e88913 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:17.069211 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:17.083651 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 242/472] 10.222.0.36 () {56 vars in 1139 bytes} [Mon Oct 31 14:44:16 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:17.084311 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042efec8-9fa3-4483-afc5-59d365e88913 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:17.100915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:17.129027 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e820db65-1612-4765-ac25-c6a89b14b5aa None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:17.165412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295e3e98-8dc3-4527-8dd9-446d290a82ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:17.166043 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295e3e98-8dc3-4527-8dd9-446d290a82ac None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:17.166323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295e3e98-8dc3-4527-8dd9-446d290a82ac None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:17.166597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295e3e98-8dc3-4527-8dd9-446d290a82ac None admin] PATH_INFO: `/v3/projects/service` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:17.174408 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295e3e98-8dc3-4527-8dd9-446d290a82ac None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:17.175844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295e3e98-8dc3-4527-8dd9-446d290a82ac None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:17.188464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-295e3e98-8dc3-4527-8dd9-446d290a82ac None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:17.188464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:17.188464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:17.188464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:17.188464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:17.188464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:17.188464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:17.188464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:17.188464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:17.188464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:17.188464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:17.188464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:17.188464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:17.188464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:17.188464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:17.189303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:17.189303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:17.189303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:17.189303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:17.189303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:17.189303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:17.189303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:17.189303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:17.189303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:17.189303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:17.189303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:17.189303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:17.189303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:17.189303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:17.189303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:17.189303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:17.189303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:17.190168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:17.190168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:17.190168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:17.190168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:17.190168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:17.190168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:17.190168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:17.190168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:17.190168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 231/473] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:17.190634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295e3e98-8dc3-4527-8dd9-446d290a82ac None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:17.201841 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042efec8-9fa3-4483-afc5-59d365e88913 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:17.239752 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2642d1d4-f52d-496b-9570-8afaf8b0deeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:17.240360 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2642d1d4-f52d-496b-9570-8afaf8b0deeb None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:17.240633 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2642d1d4-f52d-496b-9570-8afaf8b0deeb None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:17.240906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2642d1d4-f52d-496b-9570-8afaf8b0deeb None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:17.241544 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2642d1d4-f52d-496b-9570-8afaf8b0deeb None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:17.241853 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2642d1d4-f52d-496b-9570-8afaf8b0deeb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:17.243203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2642d1d4-f52d-496b-9570-8afaf8b0deeb None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:17.258786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 243/474] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:17.258786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2642d1d4-f52d-496b-9570-8afaf8b0deeb None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:17.306840 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295e3e98-8dc3-4527-8dd9-446d290a82ac None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:17.319230 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:17.335389 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:17.346204 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca86cae1-b01b-46c3-b4c3-14bc4f0d53f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:17.346877 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca86cae1-b01b-46c3-b4c3-14bc4f0d53f8 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:17.347219 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca86cae1-b01b-46c3-b4c3-14bc4f0d53f8 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:17.347643 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca86cae1-b01b-46c3-b4c3-14bc4f0d53f8 None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:17.348268 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca86cae1-b01b-46c3-b4c3-14bc4f0d53f8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:17.348571 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca86cae1-b01b-46c3-b4c3-14bc4f0d53f8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:17.349992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca86cae1-b01b-46c3-b4c3-14bc4f0d53f8 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:17.368479 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 232/475] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:17.368698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2642d1d4-f52d-496b-9570-8afaf8b0deeb None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:17.368813 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca86cae1-b01b-46c3-b4c3-14bc4f0d53f8 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:17.407319 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115d2d0a-b326-47dd-a34c-7a5159a98c72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:17.407966 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115d2d0a-b326-47dd-a34c-7a5159a98c72 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:17.408249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115d2d0a-b326-47dd-a34c-7a5159a98c72 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:17.408522 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115d2d0a-b326-47dd-a34c-7a5159a98c72 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:17.477024 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca86cae1-b01b-46c3-b4c3-14bc4f0d53f8 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:17.483953 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115d2d0a-b326-47dd-a34c-7a5159a98c72 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:17.485658 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115d2d0a-b326-47dd-a34c-7a5159a98c72 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:17.511344 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16007b80-cfe9-4520-92d2-145d9e63ae52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:17.511970 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16007b80-cfe9-4520-92d2-145d9e63ae52 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:17.512278 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16007b80-cfe9-4520-92d2-145d9e63ae52 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:17.513105 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16007b80-cfe9-4520-92d2-145d9e63ae52 None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:17.513793 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16007b80-cfe9-4520-92d2-145d9e63ae52 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:17.514096 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16007b80-cfe9-4520-92d2-145d9e63ae52 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:17.515535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16007b80-cfe9-4520-92d2-145d9e63ae52 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:17.527185 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 233/476] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:17.528124 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16007b80-cfe9-4520-92d2-145d9e63ae52 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:17.561043 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:17.568855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:17.594953 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 244/477] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:44:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 31 14:44:17.595740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115d2d0a-b326-47dd-a34c-7a5159a98c72 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:17.636839 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16007b80-cfe9-4520-92d2-145d9e63ae52 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:17.672539 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64393f9a-f3f5-420f-86fc-3659845130d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:17.673157 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64393f9a-f3f5-420f-86fc-3659845130d8 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:17.673434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64393f9a-f3f5-420f-86fc-3659845130d8 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:17.673705 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64393f9a-f3f5-420f-86fc-3659845130d8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:17.680385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64393f9a-f3f5-420f-86fc-3659845130d8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:17.681823 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64393f9a-f3f5-420f-86fc-3659845130d8 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:17.693496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-64393f9a-f3f5-420f-86fc-3659845130d8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:17.693496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:17.693496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:17.693496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:17.693496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:17.693496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:17.693496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:17.693496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:17.693496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:17.693496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:17.693496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:17.693496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:17.693496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:17.693496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:17.693496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:17.694528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:17.694528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:17.694528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:17.694528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:17.694528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:17.694528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:17.694528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:17.694528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:17.694528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:17.694528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:17.694528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:17.694528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:17.694528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:17.694528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:17.694528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:17.694528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:17.694528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:17.695670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:17.695670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:17.695670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:17.695670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:17.695670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:17.695670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:17.695670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:17.695670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:17.695670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 234/478] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:17.696247 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076b98e7-33a3-4228-a780-ccb9458f4665 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:17.696247 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076b98e7-33a3-4228-a780-ccb9458f4665 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:17.696404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076b98e7-33a3-4228-a780-ccb9458f4665 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:17.697544 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 235/479] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:17 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:17.698235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076b98e7-33a3-4228-a780-ccb9458f4665 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:17.702644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115d2d0a-b326-47dd-a34c-7a5159a98c72 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:17.737088 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fda11e-2a8a-4337-9e2d-8a9cf9c344fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:17.737854 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fda11e-2a8a-4337-9e2d-8a9cf9c344fd None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:17.737948 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fda11e-2a8a-4337-9e2d-8a9cf9c344fd None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:17.738203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fda11e-2a8a-4337-9e2d-8a9cf9c344fd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:17.738913 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fda11e-2a8a-4337-9e2d-8a9cf9c344fd None admin] RBAC: Adding query filter params (role.id=88d62c66100a4d04a45e3607135bdd0c, scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8, user.id=c8cace029a92406f850bb0a4ca07c461) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:17.739218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fda11e-2a8a-4337-9e2d-8a9cf9c344fd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:17.740669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fda11e-2a8a-4337-9e2d-8a9cf9c344fd None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:17.747287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 245/480] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:44:17 2022] GET /identity/v3/role_assignments?user.id=c8cace029a92406f850bb0a4ca07c461&scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8&role.id=88d62c66100a4d04a45e3607135bdd0c => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:17.748065 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fda11e-2a8a-4337-9e2d-8a9cf9c344fd None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:17.797990 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:17.800902 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:17.814578 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076b98e7-33a3-4228-a780-ccb9458f4665 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:17.852933 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece95321-4f27-460f-899f-33ae6070fbaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:17.853460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece95321-4f27-460f-899f-33ae6070fbaf None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:17.853739 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece95321-4f27-460f-899f-33ae6070fbaf None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:17.854010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece95321-4f27-460f-899f-33ae6070fbaf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:17.854741 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece95321-4f27-460f-899f-33ae6070fbaf None admin] RBAC: Adding query filter params (role.id=88d62c66100a4d04a45e3607135bdd0c, scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8, user.id=993d553a46044c18a4eec6e408c0969a) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:17.855048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece95321-4f27-460f-899f-33ae6070fbaf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:17.857117 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece95321-4f27-460f-899f-33ae6070fbaf None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:17.863544 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 236/481] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:44:17 2022] GET /identity/v3/role_assignments?user.id=993d553a46044c18a4eec6e408c0969a&scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8&role.id=88d62c66100a4d04a45e3607135bdd0c => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:17.864420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece95321-4f27-460f-899f-33ae6070fbaf None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:17.875768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fda11e-2a8a-4337-9e2d-8a9cf9c344fd None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:17.916418 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97285aba-5fdf-4519-ae93-5bdfe080f33b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:17.917020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97285aba-5fdf-4519-ae93-5bdfe080f33b None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:17.917228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97285aba-5fdf-4519-ae93-5bdfe080f33b None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:17.917505 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97285aba-5fdf-4519-ae93-5bdfe080f33b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:17.931587 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97285aba-5fdf-4519-ae93-5bdfe080f33b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:17.934216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97285aba-5fdf-4519-ae93-5bdfe080f33b None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:17.954234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-97285aba-5fdf-4519-ae93-5bdfe080f33b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:17.954234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:17.954234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:17.954234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:17.954234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:17.954234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:17.954234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:17.954234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:17.954234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:17.954234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:17.954234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:17.954234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:17.954234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:17.954234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:17.954234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:17.954234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:17.955198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:17.955198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:17.955198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:17.955198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:17.955198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:17.955198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:17.955198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:17.955198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:17.955198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:17.955198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:17.955198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:17.955198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:17.955198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:17.955198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:17.955198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:17.955198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:17.955198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:17.956140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:17.956140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:17.956140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:17.956590 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 246/482] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:17.957857 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb3e23b-b965-483d-82a5-8e9a234c28f3 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:17.958140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb3e23b-b965-483d-82a5-8e9a234c28f3 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:17.958410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb3e23b-b965-483d-82a5-8e9a234c28f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:17.986307 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-5bb3e23b-b965-483d-82a5-8e9a234c28f3 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:17.987532 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5bb3e23b-b965-483d-82a5-8e9a234c28f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:18.022301 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece95321-4f27-460f-899f-33ae6070fbaf None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:18.064038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368117d4-e71d-4a0d-aa83-4ece879f9ec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:18.064848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368117d4-e71d-4a0d-aa83-4ece879f9ec9 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:18.064979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368117d4-e71d-4a0d-aa83-4ece879f9ec9 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:18.065179 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368117d4-e71d-4a0d-aa83-4ece879f9ec9 None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:18.066477 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368117d4-e71d-4a0d-aa83-4ece879f9ec9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:18.066477 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368117d4-e71d-4a0d-aa83-4ece879f9ec9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:18.068164 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368117d4-e71d-4a0d-aa83-4ece879f9ec9 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:18.077822 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 247/483] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:44:18.078964 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb3e23b-b965-483d-82a5-8e9a234c28f3 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:18.082198 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 237/484] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:18.089599 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368117d4-e71d-4a0d-aa83-4ece879f9ec9 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:18.103880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:18.115914 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:18.205503 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368117d4-e71d-4a0d-aa83-4ece879f9ec9 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:18.214792 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb3e23b-b965-483d-82a5-8e9a234c28f3 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:18.253199 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a139b5ec-864a-44bf-912f-deccb8d811be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:18.254020 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a139b5ec-864a-44bf-912f-deccb8d811be None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:18.254297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a139b5ec-864a-44bf-912f-deccb8d811be None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:18.254566 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a139b5ec-864a-44bf-912f-deccb8d811be None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:18.255296 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a139b5ec-864a-44bf-912f-deccb8d811be None admin] RBAC: Adding query filter params (role.id=88d62c66100a4d04a45e3607135bdd0c, scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8, user.id=a9808dcba0b64027bb70e31d1569957d) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:18.255621 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a139b5ec-864a-44bf-912f-deccb8d811be None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:18.257061 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a139b5ec-864a-44bf-912f-deccb8d811be None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:18.268952 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 238/485] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:44:18 2022] GET /identity/v3/role_assignments?user.id=a9808dcba0b64027bb70e31d1569957d&scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8&role.id=88d62c66100a4d04a45e3607135bdd0c => generated 608 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:18.281543 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c245247-5578-4680-93e4-aa71eaee2963 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:18.282180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c245247-5578-4680-93e4-aa71eaee2963 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:18.282459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c245247-5578-4680-93e4-aa71eaee2963 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:18.282728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c245247-5578-4680-93e4-aa71eaee2963 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:18.283383 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c245247-5578-4680-93e4-aa71eaee2963 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:18.283762 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c245247-5578-4680-93e4-aa71eaee2963 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:18.285258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c245247-5578-4680-93e4-aa71eaee2963 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:18.305462 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 248/486] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:18.311273 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a139b5ec-864a-44bf-912f-deccb8d811be None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:18.410882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c245247-5578-4680-93e4-aa71eaee2963 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:18.438991 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:18.479587 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a139b5ec-864a-44bf-912f-deccb8d811be None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:18.548874 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a60b8f-1441-4e7e-b44c-e2474f975f4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:18.550223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a60b8f-1441-4e7e-b44c-e2474f975f4d None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:18.550654 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a60b8f-1441-4e7e-b44c-e2474f975f4d None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:18.551091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a60b8f-1441-4e7e-b44c-e2474f975f4d None admin] PATH_INFO: `/v3/projects/service` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:18.560661 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a60b8f-1441-4e7e-b44c-e2474f975f4d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:18.561406 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:18.562747 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a60b8f-1441-4e7e-b44c-e2474f975f4d None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:18.580279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-e6a60b8f-1441-4e7e-b44c-e2474f975f4d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:18.580279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:18.580279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:18.580279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:18.580279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:18.580279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:18.580279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:18.580279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:18.580279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:18.580279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:18.580279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:18.580279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:18.580279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:18.580279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:18.580279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:18.581359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:18.581359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:18.581359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:18.581359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:18.581359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:18.581359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:18.581359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:18.581359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:18.581359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:18.581359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:18.581359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:18.581359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:18.581359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:18.581359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:18.581359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:18.581359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:18.581359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:18.582384 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:18.582384 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:18.582384 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:18.582384 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:18.582384 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:18.582384 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:18.582384 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:18.582384 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:18.584799 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 239/487] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:18.590377 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a60b8f-1441-4e7e-b44c-e2474f975f4d None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:18.593069 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c245247-5578-4680-93e4-aa71eaee2963 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:18.648520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d95bff4-be5c-4490-9358-a6ee537a6efd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:18.649617 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d95bff4-be5c-4490-9358-a6ee537a6efd None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:18.649998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d95bff4-be5c-4490-9358-a6ee537a6efd None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:18.650367 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d95bff4-be5c-4490-9358-a6ee537a6efd None admin] PATH_INFO: `/v3/roles/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:18.664629 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d95bff4-be5c-4490-9358-a6ee537a6efd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:18.673459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-8d95bff4-be5c-4490-9358-a6ee537a6efd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:18.673459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:18.673459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:18.673459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:18.673459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:18.673459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:18.673459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:18.673459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:18.673459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:18.673459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:18.673459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:18.673459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:18.673459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:18.673459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:18.673459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:18.673459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:18.674542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:18.674542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:18.674542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:18.674542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:18.674542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:18.674542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:18.674542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:18.674542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:18.674542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:18.674542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:18.674542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:18.674542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:18.674542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:18.674542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:18.674542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:18.674542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:18.675512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:18.675512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:18.675512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:18.675512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:18.675512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:18.675512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:18.675512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:18.675512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:18.675512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:18.675512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:18.675512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:18.675512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:18.681222 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 249/488] 10.222.0.36 () {56 vars in 1102 bytes} [Mon Oct 31 14:44:18 2022] GET /identity/v3/roles/service => generated 85 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:18.682134 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d95bff4-be5c-4490-9358-a6ee537a6efd None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:18.740834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a60b8f-1441-4e7e-b44c-e2474f975f4d None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:18.785717 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d31a5ff-3fe6-4f68-8bd1-9de2eb58e5c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:18.787001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d31a5ff-3fe6-4f68-8bd1-9de2eb58e5c7 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:18.787427 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d31a5ff-3fe6-4f68-8bd1-9de2eb58e5c7 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:18.787898 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d31a5ff-3fe6-4f68-8bd1-9de2eb58e5c7 None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:18.789125 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d31a5ff-3fe6-4f68-8bd1-9de2eb58e5c7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:18.789577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d31a5ff-3fe6-4f68-8bd1-9de2eb58e5c7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:18.791256 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d31a5ff-3fe6-4f68-8bd1-9de2eb58e5c7 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:18.800550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:18.808911 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 240/489] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:18.815621 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d31a5ff-3fe6-4f68-8bd1-9de2eb58e5c7 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:18.854527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d95bff4-be5c-4490-9358-a6ee537a6efd None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:18.935259 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845765bc-ae58-406b-a52d-028be8546d24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:18.936131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845765bc-ae58-406b-a52d-028be8546d24 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:18.937642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845765bc-ae58-406b-a52d-028be8546d24 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:18.938259 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845765bc-ae58-406b-a52d-028be8546d24 None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:18.939667 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845765bc-ae58-406b-a52d-028be8546d24 None admin] RBAC: Adding query filter params (name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:18.940306 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845765bc-ae58-406b-a52d-028be8546d24 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:18.942406 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845765bc-ae58-406b-a52d-028be8546d24 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:18.953337 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d31a5ff-3fe6-4f68-8bd1-9de2eb58e5c7 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:18.964447 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 250/490] 10.222.0.36 () {56 vars in 1103 bytes} [Mon Oct 31 14:44:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:18.970019 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845765bc-ae58-406b-a52d-028be8546d24 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:18.973676 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:19.030249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e6bf04-6dc8-48b2-a4c3-e46af5cae3ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:19.030877 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e6bf04-6dc8-48b2-a4c3-e46af5cae3ea None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:19.031166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e6bf04-6dc8-48b2-a4c3-e46af5cae3ea None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:19.031442 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e6bf04-6dc8-48b2-a4c3-e46af5cae3ea None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:19.032230 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e6bf04-6dc8-48b2-a4c3-e46af5cae3ea None admin] RBAC: Adding query filter params (role.id=88d62c66100a4d04a45e3607135bdd0c, scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8, user.id=14946d97fd384e2a8bff68c4f53c2cfe) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:19.032545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e6bf04-6dc8-48b2-a4c3-e46af5cae3ea None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:19.034003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e6bf04-6dc8-48b2-a4c3-e46af5cae3ea None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:19.055987 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 241/491] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:44:18 2022] GET /identity/v3/role_assignments?user.id=14946d97fd384e2a8bff68c4f53c2cfe&scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8&role.id=88d62c66100a4d04a45e3607135bdd0c => generated 608 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:19.136430 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845765bc-ae58-406b-a52d-028be8546d24 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:19.176237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac6693d-2d20-4baa-ac04-a9d8ff77abc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:19.176887 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac6693d-2d20-4baa-ac04-a9d8ff77abc3 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:19.177170 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac6693d-2d20-4baa-ac04-a9d8ff77abc3 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:19.177446 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac6693d-2d20-4baa-ac04-a9d8ff77abc3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:19.189895 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac6693d-2d20-4baa-ac04-a9d8ff77abc3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:19.191302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac6693d-2d20-4baa-ac04-a9d8ff77abc3 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:19.207842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-6ac6693d-2d20-4baa-ac04-a9d8ff77abc3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:19.207842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:19.207842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:19.207842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:19.207842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:19.207842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:19.207842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:19.207842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:19.207842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:19.207842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:19.207842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:19.207842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:19.207842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:19.207842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:19.207842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:19.207842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:19.208807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:19.208807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:19.208807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:19.208807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:19.208807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:19.208807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:19.208807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:19.208807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:19.208807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:19.208807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:19.208807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:19.208807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:19.208807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:19.208807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:19.208807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:19.208807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:19.208807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:19.210144 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:19.210144 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:19.210144 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:19.210144 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 251/492] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:44:19.215094 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e6bf04-6dc8-48b2-a4c3-e46af5cae3ea None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:19.307913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:19.361916 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e6bf04-6dc8-48b2-a4c3-e46af5cae3ea None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:19.399544 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c828154-3851-4a1d-a959-fb2d4350c907 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:19.400244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c828154-3851-4a1d-a959-fb2d4350c907 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:19.400518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c828154-3851-4a1d-a959-fb2d4350c907 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:19.400794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c828154-3851-4a1d-a959-fb2d4350c907 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:19.401501 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c828154-3851-4a1d-a959-fb2d4350c907 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:19.401862 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c828154-3851-4a1d-a959-fb2d4350c907 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:19.403306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c828154-3851-4a1d-a959-fb2d4350c907 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:19.415638 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 242/493] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:19.421793 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac6693d-2d20-4baa-ac04-a9d8ff77abc3 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:19.482962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:19.530398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac6693d-2d20-4baa-ac04-a9d8ff77abc3 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:19.565166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1001660-a2cb-4ab1-bd77-4ca075d1469d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:19.565796 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1001660-a2cb-4ab1-bd77-4ca075d1469d None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:19.566078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1001660-a2cb-4ab1-bd77-4ca075d1469d None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:19.566352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1001660-a2cb-4ab1-bd77-4ca075d1469d None admin] PATH_INFO: `/v3/users/swift` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:19.574258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1001660-a2cb-4ab1-bd77-4ca075d1469d None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:19.576621 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1001660-a2cb-4ab1-bd77-4ca075d1469d None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:19.588249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-d1001660-a2cb-4ab1-bd77-4ca075d1469d None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 31 14:44:19.588249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:19.588249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:19.588249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:19.588249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:19.588249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:19.588249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:19.588249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:19.588249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:19.588249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:19.588249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:19.588249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:19.588249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:19.588249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:19.588249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:19.588249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:19.589159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:19.590077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:19.590077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:19.590077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:19.590077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:19.590077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:19.590077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:19.590077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:19.590077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:19.590077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 31 14:44:19.590077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:19.590077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 252/494] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:44:19 2022] GET /identity/v3/users/swift => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:19.595871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c828154-3851-4a1d-a959-fb2d4350c907 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:19.720751 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c828154-3851-4a1d-a959-fb2d4350c907 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:19.750590 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:19.764149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec030ae6-6cb4-4bf8-8332-d4edb6b29652 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:19.764762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec030ae6-6cb4-4bf8-8332-d4edb6b29652 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:19.765044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec030ae6-6cb4-4bf8-8332-d4edb6b29652 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:19.765320 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec030ae6-6cb4-4bf8-8332-d4edb6b29652 None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:19.766060 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec030ae6-6cb4-4bf8-8332-d4edb6b29652 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:19.766367 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec030ae6-6cb4-4bf8-8332-d4edb6b29652 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:19.767821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec030ae6-6cb4-4bf8-8332-d4edb6b29652 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:19.791977 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 243/495] 10.222.0.36 () {56 vars in 1135 bytes} [Mon Oct 31 14:44:19 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:19.797833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1001660-a2cb-4ab1-bd77-4ca075d1469d None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:19.916778 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1001660-a2cb-4ab1-bd77-4ca075d1469d None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:19.931433 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:19.953743 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b62eb36-cb62-4084-90a0-cbdc88bb9d9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:19.954410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b62eb36-cb62-4084-90a0-cbdc88bb9d9e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:19.954697 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b62eb36-cb62-4084-90a0-cbdc88bb9d9e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:19.954960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b62eb36-cb62-4084-90a0-cbdc88bb9d9e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:20.026493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b62eb36-cb62-4084-90a0-cbdc88bb9d9e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:20.028022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b62eb36-cb62-4084-90a0-cbdc88bb9d9e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:20.133047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 253/496] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:44:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 31 14:44:20.138977 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec030ae6-6cb4-4bf8-8332-d4edb6b29652 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:20.287723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec030ae6-6cb4-4bf8-8332-d4edb6b29652 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:20.327192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc284cc-ebb1-4b7f-a41f-3d767606878a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:20.327876 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc284cc-ebb1-4b7f-a41f-3d767606878a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:20.328154 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc284cc-ebb1-4b7f-a41f-3d767606878a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:20.328426 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc284cc-ebb1-4b7f-a41f-3d767606878a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:20.341610 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc284cc-ebb1-4b7f-a41f-3d767606878a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:20.343196 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc284cc-ebb1-4b7f-a41f-3d767606878a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:20.361636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-9fc284cc-ebb1-4b7f-a41f-3d767606878a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:20.361636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:20.361636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:20.361636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:20.361636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:20.361636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:20.361636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:20.361636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:20.361636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:20.361636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:20.361636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:20.361636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:20.361636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:20.361636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:20.361636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:20.361636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:20.362603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:20.362603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:20.362603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:20.362603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:20.362603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:20.362603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:20.362603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:20.362603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:20.362603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:20.362603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:20.362603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:20.362603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:20.362603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:20.362603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:20.362603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:20.362603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:20.362603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:20.363580 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:20.363580 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:20.363580 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:20.363580 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 244/497] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:44:20.369916 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b62eb36-cb62-4084-90a0-cbdc88bb9d9e None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:20.395965 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:20.442360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384d03c6-6196-4ace-a919-724d00c8ed7a None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:20.443594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384d03c6-6196-4ace-a919-724d00c8ed7a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:20.443873 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384d03c6-6196-4ace-a919-724d00c8ed7a None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:20.445343 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 245/498] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:20 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:44:20.446190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e11a6b-f319-46f0-81dc-4771b8adec76 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:20.446479 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e11a6b-f319-46f0-81dc-4771b8adec76 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:20.446745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e11a6b-f319-46f0-81dc-4771b8adec76 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:20.448748 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 246/499] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:20 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:20.454271 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e1a883-3a5d-46f4-a81d-edd1e3844d17 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:20.454585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e1a883-3a5d-46f4-a81d-edd1e3844d17 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:20.454866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e1a883-3a5d-46f4-a81d-edd1e3844d17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:20.475299 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-54e1a883-3a5d-46f4-a81d-edd1e3844d17 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:20.476521 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-54e1a883-3a5d-46f4-a81d-edd1e3844d17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:20.508374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:20.517702 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b62eb36-cb62-4084-90a0-cbdc88bb9d9e None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:20.554071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 247/500] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:44:20.555849 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ac7c80-b3b5-4eac-b064-dcd164f90de5 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:20.556359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ac7c80-b3b5-4eac-b064-dcd164f90de5 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:20.556840 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ac7c80-b3b5-4eac-b064-dcd164f90de5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:20.571589 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a40d010-960f-42a5-b379-322874bb21d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:20.572358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a40d010-960f-42a5-b379-322874bb21d5 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:20.572809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a40d010-960f-42a5-b379-322874bb21d5 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:20.573088 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a40d010-960f-42a5-b379-322874bb21d5 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:20.573787 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a40d010-960f-42a5-b379-322874bb21d5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:20.574090 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a40d010-960f-42a5-b379-322874bb21d5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:20.576195 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a40d010-960f-42a5-b379-322874bb21d5 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:20.578120 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-56ac7c80-b3b5-4eac-b064-dcd164f90de5 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:20.579512 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-56ac7c80-b3b5-4eac-b064-dcd164f90de5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:20.595364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 254/501] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:20.601104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a40d010-960f-42a5-b379-322874bb21d5 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:20.698102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 248/502] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:44:20.730032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:20.759991 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a40d010-960f-42a5-b379-322874bb21d5 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:20.795787 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece79947-54cf-40e3-be66-849755e33abc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:20.796545 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece79947-54cf-40e3-be66-849755e33abc None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:20.796879 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece79947-54cf-40e3-be66-849755e33abc None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:20.797150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece79947-54cf-40e3-be66-849755e33abc None admin] PATH_INFO: `/v3/projects/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:20.803880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece79947-54cf-40e3-be66-849755e33abc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:20.805427 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece79947-54cf-40e3-be66-849755e33abc None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:20.817615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-ece79947-54cf-40e3-be66-849755e33abc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:20.817615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:20.817615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:20.817615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:20.817615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:20.817615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:20.817615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:20.817615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:20.817615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:20.817615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:20.817615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:20.817615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:20.817615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:20.817615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:20.817615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:20.818922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:20.818922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:20.818922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:20.818922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:20.818922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:20.818922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:20.818922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:20.818922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:20.818922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:20.818922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:20.818922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:20.818922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:20.818922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:20.818922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:20.818922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:20.818922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:20.818922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:20.820387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:20.820387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:20.820387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:20.820387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:20.820387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:20.820387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:20.820387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:20.820387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:20.820387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 255/503] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:20.824402 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ac7c80-b3b5-4eac-b064-dcd164f90de5 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:20.942058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece79947-54cf-40e3-be66-849755e33abc None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:20.968680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:20.983854 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ac7c80-b3b5-4eac-b064-dcd164f90de5 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:21.040133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6b86f8-4821-44a2-8efb-e0ed229e6280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:21.041208 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6b86f8-4821-44a2-8efb-e0ed229e6280 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:21.042394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6b86f8-4821-44a2-8efb-e0ed229e6280 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:21.042983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6b86f8-4821-44a2-8efb-e0ed229e6280 None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:21.044983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6b86f8-4821-44a2-8efb-e0ed229e6280 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:21.045489 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6b86f8-4821-44a2-8efb-e0ed229e6280 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:21.047257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6b86f8-4821-44a2-8efb-e0ed229e6280 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:21.071665 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 249/504] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:21.071665 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6b86f8-4821-44a2-8efb-e0ed229e6280 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:21.118341 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece79947-54cf-40e3-be66-849755e33abc None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:21.160347 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b49b69-ad35-4c85-980f-481d72fa7681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:21.160973 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b49b69-ad35-4c85-980f-481d72fa7681 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:21.161248 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b49b69-ad35-4c85-980f-481d72fa7681 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:21.161515 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b49b69-ad35-4c85-980f-481d72fa7681 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:21.165618 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:21.179601 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b49b69-ad35-4c85-980f-481d72fa7681 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:21.187977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-45b49b69-ad35-4c85-980f-481d72fa7681 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:44:21.187977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:21.187977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:21.187977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:21.187977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:21.187977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:21.187977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:21.187977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:21.187977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:21.187977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:21.187977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:21.187977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:21.187977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:21.187977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:21.187977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:21.187977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:21.189150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:21.189150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:21.189150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:21.189150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:21.189150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:21.189150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:21.189150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:21.189150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:21.189150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:21.189150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:21.189150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:21.189150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:21.189150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:21.189150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:21.189150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:21.189150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:21.190184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:21.190184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:21.190184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:21.190184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:21.190184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:21.190184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:21.190184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:21.190184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:21.190184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:21.190184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:21.190184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:44:21.190184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:21.193092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 256/505] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:44:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:21.194714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b49b69-ad35-4c85-980f-481d72fa7681 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:21.212013 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6b86f8-4821-44a2-8efb-e0ed229e6280 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:21.250862 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1d4294-b52e-4fd5-94b8-a3369af9fd4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:21.251677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1d4294-b52e-4fd5-94b8-a3369af9fd4a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:21.252047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1d4294-b52e-4fd5-94b8-a3369af9fd4a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:21.252416 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1d4294-b52e-4fd5-94b8-a3369af9fd4a None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:21.253088 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1d4294-b52e-4fd5-94b8-a3369af9fd4a None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:21.254597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1d4294-b52e-4fd5-94b8-a3369af9fd4a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:21.269346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-4e1d4294-b52e-4fd5-94b8-a3369af9fd4a None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 31 14:44:21.269346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:21.269346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:21.269346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:21.269346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:21.269346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:21.269346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:21.269346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:21.269346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 31 14:44:21.269346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:21.269346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:21.269346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:21.269346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:21.269346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:21.269346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:21.269346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:21.269346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:21.270515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:21.270515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:21.270515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:21.270515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:21.270515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:21.270515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:21.270515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:21.270515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:21.270515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:21.270515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:21.270515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:21.270515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:21.270515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:21.270515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:21.270515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:21.270515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:21.271440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:21.271440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:21.271440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:21.271440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:21.271440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:21.271440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:21.271440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:21.271440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:21.271440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:21.271440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:21.271440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:21.271440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:21.271440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 31 14:44:21.271440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:21.274621 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 250/506] 10.222.0.36 () {56 vars in 1129 bytes} [Mon Oct 31 14:44:21 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:21.275629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1d4294-b52e-4fd5-94b8-a3369af9fd4a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:21.300811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:21.331324 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b49b69-ad35-4c85-980f-481d72fa7681 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:21.369072 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef4b32b-d859-4d3b-a9ca-55f0e4902c85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:21.369673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef4b32b-d859-4d3b-a9ca-55f0e4902c85 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:21.369949 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef4b32b-d859-4d3b-a9ca-55f0e4902c85 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:21.370218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef4b32b-d859-4d3b-a9ca-55f0e4902c85 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:21.370930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef4b32b-d859-4d3b-a9ca-55f0e4902c85 None admin] RBAC: Adding query filter params (role.id=88d62c66100a4d04a45e3607135bdd0c, scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8, user.id=f4ae5e20cc5b40ccbdb951f908f5080d) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:21.371367 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef4b32b-d859-4d3b-a9ca-55f0e4902c85 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:21.372996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef4b32b-d859-4d3b-a9ca-55f0e4902c85 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:21.387594 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 257/507] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:44:21 2022] GET /identity/v3/role_assignments?user.id=f4ae5e20cc5b40ccbdb951f908f5080d&scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8&role.id=88d62c66100a4d04a45e3607135bdd0c => generated 608 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:21.387594 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef4b32b-d859-4d3b-a9ca-55f0e4902c85 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:21.401405 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1d4294-b52e-4fd5-94b8-a3369af9fd4a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:21.442290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807761c1-d974-419f-9257-c6088095d362 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:21.442964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807761c1-d974-419f-9257-c6088095d362 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:21.443316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807761c1-d974-419f-9257-c6088095d362 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:21.443755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807761c1-d974-419f-9257-c6088095d362 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:21.444509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807761c1-d974-419f-9257-c6088095d362 None admin] RBAC: Adding query filter params (name=admin) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:21.445041 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807761c1-d974-419f-9257-c6088095d362 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:21.446652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807761c1-d974-419f-9257-c6088095d362 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:21.462260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 251/508] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:44:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:21.463451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:21.464990 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235fb27c-7a99-475e-8ac0-4a70593981da None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:21.465276 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235fb27c-7a99-475e-8ac0-4a70593981da None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:21.466300 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235fb27c-7a99-475e-8ac0-4a70593981da None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:21.473077 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 252/509] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:21 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:21.474118 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235fb27c-7a99-475e-8ac0-4a70593981da None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:21.516867 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef4b32b-d859-4d3b-a9ca-55f0e4902c85 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:21.572461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c7d94b-988f-4cc1-9638-41efa1c7b4d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:21.574312 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c7d94b-988f-4cc1-9638-41efa1c7b4d6 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:21.575370 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c7d94b-988f-4cc1-9638-41efa1c7b4d6 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:21.575864 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c7d94b-988f-4cc1-9638-41efa1c7b4d6 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:21.576770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c7d94b-988f-4cc1-9638-41efa1c7b4d6 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:21.577324 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c7d94b-988f-4cc1-9638-41efa1c7b4d6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:21.578921 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c7d94b-988f-4cc1-9638-41efa1c7b4d6 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:21.585916 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 258/510] 10.222.0.36 () {56 vars in 1124 bytes} [Mon Oct 31 14:44:21 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:21.589969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c45e31-1cbd-4374-bb6a-429ccbab8237 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:21.590947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c45e31-1cbd-4374-bb6a-429ccbab8237 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:21.591316 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c45e31-1cbd-4374-bb6a-429ccbab8237 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:21.600333 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:21.612803 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-35c45e31-1cbd-4374-bb6a-429ccbab8237 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:21.614458 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-35c45e31-1cbd-4374-bb6a-429ccbab8237 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:21.631111 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235fb27c-7a99-475e-8ac0-4a70593981da None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:21.676907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6bb8a6-e671-4cb8-be57-b1c4139f288f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:21.678017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6bb8a6-e671-4cb8-be57-b1c4139f288f None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:21.678506 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6bb8a6-e671-4cb8-be57-b1c4139f288f None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:21.679003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6bb8a6-e671-4cb8-be57-b1c4139f288f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:21.691912 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6bb8a6-e671-4cb8-be57-b1c4139f288f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:21.693908 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6bb8a6-e671-4cb8-be57-b1c4139f288f None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:21.707587 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 259/511] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:44:21.707587 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c45e31-1cbd-4374-bb6a-429ccbab8237 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:21.721270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-0a6bb8a6-e671-4cb8-be57-b1c4139f288f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:21.721270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:21.721270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:21.721270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:21.721270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:21.721270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:21.721270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:21.721270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:21.721270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:21.721270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:21.721270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:21.721270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:21.721270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:21.721270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:21.721270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:21.721270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:21.722568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:21.722568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:21.722568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:21.722568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:21.722568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:21.722568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:21.722568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:21.722568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:21.722568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:21.722568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:21.722568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:21.722568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:21.722568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:21.722568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:21.722568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:21.722568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:21.722568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:21.723453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:21.723453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:21.723453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:21.733760 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 253/512] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:21.740206 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6bb8a6-e671-4cb8-be57-b1c4139f288f None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:21.761403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:21.821327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c45e31-1cbd-4374-bb6a-429ccbab8237 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:21.862474 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd57983d-0391-4a2b-8801-83ec2d0ef342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:21.863233 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd57983d-0391-4a2b-8801-83ec2d0ef342 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:21.863518 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd57983d-0391-4a2b-8801-83ec2d0ef342 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:21.863904 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd57983d-0391-4a2b-8801-83ec2d0ef342 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:21.864635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd57983d-0391-4a2b-8801-83ec2d0ef342 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:21.864950 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd57983d-0391-4a2b-8801-83ec2d0ef342 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:21.866722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd57983d-0391-4a2b-8801-83ec2d0ef342 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:21.875362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 260/513] 10.222.0.36 () {56 vars in 1124 bytes} [Mon Oct 31 14:44:21 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:21.947348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6bb8a6-e671-4cb8-be57-b1c4139f288f None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:21.984416 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15566a5a-6e5f-4b3a-b74d-517a7025b7a4 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:21.984946 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15566a5a-6e5f-4b3a-b74d-517a7025b7a4 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:21.984946 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15566a5a-6e5f-4b3a-b74d-517a7025b7a4 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:21.986529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 261/514] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:44:21.991787 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06538a41-f81a-4253-9689-cbc76c0f4d05 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:21.992072 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06538a41-f81a-4253-9689-cbc76c0f4d05 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:21.992344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06538a41-f81a-4253-9689-cbc76c0f4d05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:21.995292 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:22.009634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbeb493-e3bd-4eef-b0cb-6a28a729ee7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:22.010266 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbeb493-e3bd-4eef-b0cb-6a28a729ee7e None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:22.010543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbeb493-e3bd-4eef-b0cb-6a28a729ee7e None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:22.010818 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbeb493-e3bd-4eef-b0cb-6a28a729ee7e None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:22.011482 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbeb493-e3bd-4eef-b0cb-6a28a729ee7e None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:22.011813 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbeb493-e3bd-4eef-b0cb-6a28a729ee7e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:22.014135 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbeb493-e3bd-4eef-b0cb-6a28a729ee7e None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:22.026900 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-06538a41-f81a-4253-9689-cbc76c0f4d05 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:22.028216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-06538a41-f81a-4253-9689-cbc76c0f4d05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:22.041639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 254/515] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:22.046814 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbeb493-e3bd-4eef-b0cb-6a28a729ee7e None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:22.125503 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 262/516] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:44:22.126322 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06538a41-f81a-4253-9689-cbc76c0f4d05 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:22.179707 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbeb493-e3bd-4eef-b0cb-6a28a729ee7e None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:22.205827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:22.230342 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9a673c-04a6-40d5-aa9a-60b4139d671d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:22.231931 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9a673c-04a6-40d5-aa9a-60b4139d671d None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:22.232650 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9a673c-04a6-40d5-aa9a-60b4139d671d None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:22.234016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9a673c-04a6-40d5-aa9a-60b4139d671d None admin] PATH_INFO: `/v3/users/nova` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:22.243437 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9a673c-04a6-40d5-aa9a-60b4139d671d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:22.245179 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9a673c-04a6-40d5-aa9a-60b4139d671d None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:22.259332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-ab9a673c-04a6-40d5-aa9a-60b4139d671d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 31 14:44:22.259332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:22.259332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:22.259332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:22.259332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:22.259332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:22.259332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:22.259332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:22.259332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:22.259332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:22.259332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:22.259332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:22.259332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:22.259332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:22.259332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:22.259332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:22.260535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:22.261648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:22.261648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:22.261648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:22.261648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:22.261648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:22.261648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:22.261648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:22.261648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:22.261648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 31 14:44:22.261648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:22.261648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 255/517] 10.222.0.36 () {56 vars in 1093 bytes} [Mon Oct 31 14:44:22 2022] GET /identity/v3/users/nova => generated 82 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:44:22.266831 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9a673c-04a6-40d5-aa9a-60b4139d671d None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:22.273189 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06538a41-f81a-4253-9689-cbc76c0f4d05 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:22.310533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46955a6-ec95-45a5-802a-68790ab711f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:22.311513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46955a6-ec95-45a5-802a-68790ab711f8 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:22.311987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46955a6-ec95-45a5-802a-68790ab711f8 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:22.312413 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46955a6-ec95-45a5-802a-68790ab711f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:22.323658 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46955a6-ec95-45a5-802a-68790ab711f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:22.325590 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46955a6-ec95-45a5-802a-68790ab711f8 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:22.342546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-e46955a6-ec95-45a5-802a-68790ab711f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:22.342546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:22.342546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:22.342546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:22.342546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:22.342546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:22.342546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:22.342546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:22.342546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:22.342546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:22.342546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:22.342546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:22.342546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:22.342546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:22.342546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:22.342546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:22.343903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:22.343903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:22.343903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:22.343903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:22.343903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:22.343903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:22.343903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:22.343903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:22.343903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:22.343903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:22.343903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:22.343903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:22.343903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:22.343903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:22.343903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:22.343903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:22.343903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:22.345222 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:22.345222 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:22.345222 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:22.347034 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 263/518] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:22.353591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46955a6-ec95-45a5-802a-68790ab711f8 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:22.373126 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:22.438037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9a673c-04a6-40d5-aa9a-60b4139d671d None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:22.488777 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e54d36-db1c-4a2f-9eb6-2bc91b38ab4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:22.489992 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:22.490479 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e54d36-db1c-4a2f-9eb6-2bc91b38ab4b None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:22.492150 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e54d36-db1c-4a2f-9eb6-2bc91b38ab4b None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:22.492744 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e54d36-db1c-4a2f-9eb6-2bc91b38ab4b None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:22.493755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e54d36-db1c-4a2f-9eb6-2bc91b38ab4b None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:22.494237 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e54d36-db1c-4a2f-9eb6-2bc91b38ab4b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:22.494449 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46955a6-ec95-45a5-802a-68790ab711f8 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:22.497183 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e54d36-db1c-4a2f-9eb6-2bc91b38ab4b None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:22.524792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 256/519] 10.222.0.36 () {56 vars in 1133 bytes} [Mon Oct 31 14:44:22 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:22.529881 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e54d36-db1c-4a2f-9eb6-2bc91b38ab4b None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:22.539517 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1ed80d-1d46-4fe9-a042-4ff12b5ba400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:22.540254 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1ed80d-1d46-4fe9-a042-4ff12b5ba400 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:22.540540 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1ed80d-1d46-4fe9-a042-4ff12b5ba400 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:22.540812 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1ed80d-1d46-4fe9-a042-4ff12b5ba400 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:22.541622 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1ed80d-1d46-4fe9-a042-4ff12b5ba400 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:22.541732 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1ed80d-1d46-4fe9-a042-4ff12b5ba400 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:22.543684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1ed80d-1d46-4fe9-a042-4ff12b5ba400 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:22.556408 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 264/520] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:22.557342 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1ed80d-1d46-4fe9-a042-4ff12b5ba400 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:22.664054 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1ed80d-1d46-4fe9-a042-4ff12b5ba400 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:22.672778 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e54d36-db1c-4a2f-9eb6-2bc91b38ab4b None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:22.686415 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:22.712706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8bd76e-9864-4b30-af2f-128fcfd4b7e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:22.713119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe864cf-b304-4cf8-9784-41ca8b70c09b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:22.713743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe864cf-b304-4cf8-9784-41ca8b70c09b None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:22.714021 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe864cf-b304-4cf8-9784-41ca8b70c09b None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:22.714291 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe864cf-b304-4cf8-9784-41ca8b70c09b None admin] PATH_INFO: `/v3/services/network` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:22.714521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8bd76e-9864-4b30-af2f-128fcfd4b7e9 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:22.714869 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe864cf-b304-4cf8-9784-41ca8b70c09b None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:22.715734 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8bd76e-9864-4b30-af2f-128fcfd4b7e9 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:22.716200 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8bd76e-9864-4b30-af2f-128fcfd4b7e9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:22.717131 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe864cf-b304-4cf8-9784-41ca8b70c09b None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:22.726550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-9fe864cf-b304-4cf8-9784-41ca8b70c09b None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 31 14:44:22.726550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:22.726550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:22.726550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:22.726550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:22.726550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:22.726550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:22.726550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:22.726550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 31 14:44:22.726550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:22.726550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:22.726550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:22.726550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:22.726550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:22.726550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:22.726550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:22.726550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:22.727969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:22.727969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:22.727969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:22.727969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:22.727969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:22.727969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:22.727969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:22.727969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:22.727969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:22.727969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:22.727969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:22.727969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:22.727969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:22.727969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:22.727969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:22.727969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:22.728829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:22.728829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:22.728829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:22.728829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:22.728829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:22.728829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:22.728829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:22.728829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:22.728829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:22.728829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:22.728829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:22.728829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:22.728829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 31 14:44:22.728829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:22.731922 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 257/521] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:22 2022] GET /identity/v3/services/network => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:22.732682 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8bd76e-9864-4b30-af2f-128fcfd4b7e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:22.732807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe864cf-b304-4cf8-9784-41ca8b70c09b None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:22.734643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8bd76e-9864-4b30-af2f-128fcfd4b7e9 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:22.755297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-da8bd76e-9864-4b30-af2f-128fcfd4b7e9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:22.755297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:22.755297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:22.755297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:22.755297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:22.755297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:22.755297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:22.755297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:22.755297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:22.755297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:22.755297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:22.755297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:22.755297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:22.755297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:22.755297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:22.755297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:22.756512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:22.756512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:22.756512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:22.756512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:22.756512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:22.756512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:22.756512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:22.756512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:22.756512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:22.756512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:22.756512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:22.756512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:22.756512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:22.756512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:22.756512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:22.756512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:22.756512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:22.757468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:22.757468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:22.757468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:22.764638 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 265/522] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:22.764638 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8bd76e-9864-4b30-af2f-128fcfd4b7e9 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:22.783058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:22.864696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe864cf-b304-4cf8-9784-41ca8b70c09b None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:22.890270 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8bd76e-9864-4b30-af2f-128fcfd4b7e9 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:22.909290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c65571b-86a3-4c50-8189-d0e145823def None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:22.910273 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c65571b-86a3-4c50-8189-d0e145823def None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:22.910554 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c65571b-86a3-4c50-8189-d0e145823def None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:22.910829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c65571b-86a3-4c50-8189-d0e145823def None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:22.911579 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c65571b-86a3-4c50-8189-d0e145823def None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:22.914029 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c65571b-86a3-4c50-8189-d0e145823def None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:22.932618 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c77d4cb-227c-4e35-92ea-4a090cb069c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:22.933244 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c77d4cb-227c-4e35-92ea-4a090cb069c4 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:22.933521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c77d4cb-227c-4e35-92ea-4a090cb069c4 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:22.934449 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c77d4cb-227c-4e35-92ea-4a090cb069c4 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:22.935085 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c77d4cb-227c-4e35-92ea-4a090cb069c4 None admin] RBAC: Adding query filter params (name=network) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:22.935394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c77d4cb-227c-4e35-92ea-4a090cb069c4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:22.936793 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c77d4cb-227c-4e35-92ea-4a090cb069c4 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:22.941616 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 266/523] 10.222.0.36 () {56 vars in 1112 bytes} [Mon Oct 31 14:44:22 2022] GET /identity/v3/services?name=network => generated 124 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:22.946367 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c77d4cb-227c-4e35-92ea-4a090cb069c4 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:22.953946 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 258/524] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:44:22 2022] POST /identity/v3/users => generated 254 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:22.954914 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c65571b-86a3-4c50-8189-d0e145823def None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:23.021341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:23.048949 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:23.084587 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c77d4cb-227c-4e35-92ea-4a090cb069c4 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:23.132170 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5057b95-2c63-43ec-873f-10ca890729e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:23.133313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5057b95-2c63-43ec-873f-10ca890729e2 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:23.133816 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5057b95-2c63-43ec-873f-10ca890729e2 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:23.134300 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5057b95-2c63-43ec-873f-10ca890729e2 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:23.135151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5057b95-2c63-43ec-873f-10ca890729e2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:23.135687 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5057b95-2c63-43ec-873f-10ca890729e2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:23.137589 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5057b95-2c63-43ec-873f-10ca890729e2 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:23.151866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 267/525] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:23.165663 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5057b95-2c63-43ec-873f-10ca890729e2 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:23.197929 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c65571b-86a3-4c50-8189-d0e145823def None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:23.254202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc9c6c1-76c0-4be8-970a-47970c47e717 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:23.255240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc9c6c1-76c0-4be8-970a-47970c47e717 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:23.255530 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc9c6c1-76c0-4be8-970a-47970c47e717 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:23.256106 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc9c6c1-76c0-4be8-970a-47970c47e717 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:23.256752 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc9c6c1-76c0-4be8-970a-47970c47e717 None admin] RBAC: Adding query filter params (type=network) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:23.257317 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc9c6c1-76c0-4be8-970a-47970c47e717 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:23.259119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc9c6c1-76c0-4be8-970a-47970c47e717 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:23.267704 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 259/526] 10.222.0.36 () {56 vars in 1112 bytes} [Mon Oct 31 14:44:22 2022] GET /identity/v3/services?type=network => generated 124 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:23.306250 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5057b95-2c63-43ec-873f-10ca890729e2 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:23.346423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc98fd11-135c-40e2-8bf4-490a741c8d1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:23.347058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc98fd11-135c-40e2-8bf4-490a741c8d1b None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:23.347331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc98fd11-135c-40e2-8bf4-490a741c8d1b None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:23.347690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc98fd11-135c-40e2-8bf4-490a741c8d1b None admin] PATH_INFO: `/v3/projects/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:23.349794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:23.356599 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc98fd11-135c-40e2-8bf4-490a741c8d1b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:23.358532 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc98fd11-135c-40e2-8bf4-490a741c8d1b None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:23.371942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-dc98fd11-135c-40e2-8bf4-490a741c8d1b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:23.371942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:23.371942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:23.371942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:23.371942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:23.371942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:23.371942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:23.371942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:23.371942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:23.371942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:23.371942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:23.371942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:23.371942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:23.371942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:23.371942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:23.372870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:23.372870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:23.372870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:23.372870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:23.372870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:23.372870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:23.372870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:23.372870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:23.372870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:23.372870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:23.372870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:23.372870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:23.372870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:23.372870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:23.372870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:23.372870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:23.372870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:23.373829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:23.373829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:23.373829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:23.373829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:23.373829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:23.373829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:23.373829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:23.373829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:23.373829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 268/527] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:23.382204 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc9c6c1-76c0-4be8-970a-47970c47e717 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:23.494602 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc9c6c1-76c0-4be8-970a-47970c47e717 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:23.496714 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:23.532015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c02f4e-3739-46cb-9018-c520c69caded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:23.532620 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c02f4e-3739-46cb-9018-c520c69caded None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:23.532896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c02f4e-3739-46cb-9018-c520c69caded None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:23.533216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c02f4e-3739-46cb-9018-c520c69caded None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:23.533905 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c02f4e-3739-46cb-9018-c520c69caded None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:23.534712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c02f4e-3739-46cb-9018-c520c69caded None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:23.537009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c02f4e-3739-46cb-9018-c520c69caded None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:23.548293 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 260/528] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:23.553622 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc98fd11-135c-40e2-8bf4-490a741c8d1b None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:23.662004 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc98fd11-135c-40e2-8bf4-490a741c8d1b None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:23.699202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dccb61-f945-491d-a2f8-83d9dcaddfd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:23.700061 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dccb61-f945-491d-a2f8-83d9dcaddfd9 None admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:23.700339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dccb61-f945-491d-a2f8-83d9dcaddfd9 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:23.700610 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dccb61-f945-491d-a2f8-83d9dcaddfd9 None admin] PATH_INFO: `/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/c8cace029a92406f850bb0a4ca07c461/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:23.702571 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10dccb61-f945-491d-a2f8-83d9dcaddfd9 None admin] RBAC: Authorizing `identity:create_grant(project_id=4118e33b099f4cf0a3e3a46b211ba1d8, user_id=c8cace029a92406f850bb0a4ca07c461, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:23.704037 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10dccb61-f945-491d-a2f8-83d9dcaddfd9 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:23.721908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 269/529] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:44:23 2022] PUT /identity/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/c8cace029a92406f850bb0a4ca07c461/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:44:23.925030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b738a9e9-30e1-4993-94ab-877b60b78052 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:23.925525 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b738a9e9-30e1-4993-94ab-877b60b78052 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:23.925525 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b738a9e9-30e1-4993-94ab-877b60b78052 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:23.927078 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 261/530] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:23.932435 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930ac4a8-6249-4b39-9896-75d1dad4be1b None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:23.932723 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930ac4a8-6249-4b39-9896-75d1dad4be1b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:23.932999 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930ac4a8-6249-4b39-9896-75d1dad4be1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:23.954425 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-930ac4a8-6249-4b39-9896-75d1dad4be1b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:23.955737 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-930ac4a8-6249-4b39-9896-75d1dad4be1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:24.012936 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:24.033466 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 270/531] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:44:24.372344 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b738a9e9-30e1-4993-94ab-877b60b78052 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:24.469315 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07b3427-448f-4822-85e2-c38335d84fe1 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:24.469827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07b3427-448f-4822-85e2-c38335d84fe1 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:24.469827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07b3427-448f-4822-85e2-c38335d84fe1 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:24.471430 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 271/532] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:24.476092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1650ed-60de-49ba-b697-1c3db46dd730 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:24.476367 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1650ed-60de-49ba-b697-1c3db46dd730 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:24.476637 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1650ed-60de-49ba-b697-1c3db46dd730 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:24.499536 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b738a9e9-30e1-4993-94ab-877b60b78052 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:24.514029 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-7f1650ed-60de-49ba-b697-1c3db46dd730 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:24.515214 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7f1650ed-60de-49ba-b697-1c3db46dd730 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:24.571035 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b4183c-f052-44d5-8238-b2cea8e812c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:24.571721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b4183c-f052-44d5-8238-b2cea8e812c6 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:24.572000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b4183c-f052-44d5-8238-b2cea8e812c6 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:24.572271 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b4183c-f052-44d5-8238-b2cea8e812c6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:24.598498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b4183c-f052-44d5-8238-b2cea8e812c6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:24.606995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-01b4183c-f052-44d5-8238-b2cea8e812c6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:44:24.606995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:24.606995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:24.606995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:24.606995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:24.606995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:24.606995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:24.606995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:24.606995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:24.606995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:24.606995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:24.606995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:24.606995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:24.606995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:24.606995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:24.606995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:24.607979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:24.607979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:24.607979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:24.607979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:24.607979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:24.607979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:24.607979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:24.607979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:24.607979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:24.607979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:24.607979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:24.607979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:24.607979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:24.607979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:24.607979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:24.607979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:24.608820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:24.608820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:24.608820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:24.608820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:24.608820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:24.608820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:24.608820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:24.608820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:24.608820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:24.608820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:24.608820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:44:24.608820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:24.615027 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 262/533] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:44:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:24.615907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b4183c-f052-44d5-8238-b2cea8e812c6 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:24.640677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:24.644813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 272/534] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 31 14:44:24.750706 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b4183c-f052-44d5-8238-b2cea8e812c6 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:24.795939 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6605e2-453d-483d-b35c-a6be39515e4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:24.796880 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6605e2-453d-483d-b35c-a6be39515e4f None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:24.797360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6605e2-453d-483d-b35c-a6be39515e4f None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:24.797832 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6605e2-453d-483d-b35c-a6be39515e4f None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:24.798691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6605e2-453d-483d-b35c-a6be39515e4f None admin] RBAC: Adding query filter params (name=admin) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:24.799203 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6605e2-453d-483d-b35c-a6be39515e4f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:24.800859 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6605e2-453d-483d-b35c-a6be39515e4f None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:24.815420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 263/535] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:44:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:24.821554 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1650ed-60de-49ba-b697-1c3db46dd730 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:24.980348 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1650ed-60de-49ba-b697-1c3db46dd730 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:24.994963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6605e2-453d-483d-b35c-a6be39515e4f None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:25.001789 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:25.026842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecd7ed3-af00-4e22-b3d9-936b1812cd2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:25.027782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecd7ed3-af00-4e22-b3d9-936b1812cd2a None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:25.028067 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecd7ed3-af00-4e22-b3d9-936b1812cd2a None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:25.028331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecd7ed3-af00-4e22-b3d9-936b1812cd2a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:25.048589 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecd7ed3-af00-4e22-b3d9-936b1812cd2a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:25.050043 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecd7ed3-af00-4e22-b3d9-936b1812cd2a None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:25.080947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-eecd7ed3-af00-4e22-b3d9-936b1812cd2a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:25.080947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:25.080947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:25.080947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:25.080947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:25.080947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:25.080947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:25.080947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:25.080947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:25.080947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:25.080947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:25.080947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:25.080947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:25.080947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:25.080947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:25.080947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:25.081928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:25.081928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:25.081928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:25.081928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:25.081928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:25.081928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:25.081928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:25.081928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:25.081928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:25.081928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:25.081928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:25.081928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:25.081928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:25.081928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:25.081928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:25.081928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:25.081928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:25.082787 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:25.082787 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:25.082787 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:25.082787 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 273/536] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:25.087921 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecd7ed3-af00-4e22-b3d9-936b1812cd2a None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:25.150246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:25.209456 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6605e2-453d-483d-b35c-a6be39515e4f None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:25.270539 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecd7ed3-af00-4e22-b3d9-936b1812cd2a None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:25.275712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886f3929-033a-40f3-a864-f4bdff3ec140 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:25.278641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886f3929-033a-40f3-a864-f4bdff3ec140 None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:25.281614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886f3929-033a-40f3-a864-f4bdff3ec140 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:25.282113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886f3929-033a-40f3-a864-f4bdff3ec140 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:25.282987 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886f3929-033a-40f3-a864-f4bdff3ec140 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:25.284616 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886f3929-033a-40f3-a864-f4bdff3ec140 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:25.314226 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 264/537] 10.222.0.36 () {60 vars in 1141 bytes} [Mon Oct 31 14:44:24 2022] POST /identity/v3/services => generated 251 bytes in 320 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:25.349033 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea4270e-cd35-4171-b924-41f305b1406e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:25.350063 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea4270e-cd35-4171-b924-41f305b1406e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:25.350608 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea4270e-cd35-4171-b924-41f305b1406e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:25.351097 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea4270e-cd35-4171-b924-41f305b1406e None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:25.352082 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea4270e-cd35-4171-b924-41f305b1406e None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:25.352599 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea4270e-cd35-4171-b924-41f305b1406e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:25.354320 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea4270e-cd35-4171-b924-41f305b1406e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:25.379210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 274/538] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:25.385062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886f3929-033a-40f3-a864-f4bdff3ec140 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:25.516960 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886f3929-033a-40f3-a864-f4bdff3ec140 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:25.567714 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c807f0ce-28af-4d00-9258-9772c4d91c32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:25.568361 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c807f0ce-28af-4d00-9258-9772c4d91c32 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:25.568637 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c807f0ce-28af-4d00-9258-9772c4d91c32 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:25.568908 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c807f0ce-28af-4d00-9258-9772c4d91c32 None admin] PATH_INFO: `/v3/users/swift` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:25.572373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:25.581973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c807f0ce-28af-4d00-9258-9772c4d91c32 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:25.583486 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c807f0ce-28af-4d00-9258-9772c4d91c32 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:25.598104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-c807f0ce-28af-4d00-9258-9772c4d91c32 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 31 14:44:25.598104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:25.598104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:25.598104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:25.598104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:25.598104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:25.598104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:25.598104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:25.598104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:25.598104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:25.598104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:25.598104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:25.598104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:25.598104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:25.598104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:25.598104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:25.599074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:25.600116 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:25.600116 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:25.600116 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:25.600116 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:25.600116 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:25.600116 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:25.600116 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:25.600116 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:25.600116 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 31 14:44:25.600116 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:25.600116 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 265/539] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:44:25 2022] GET /identity/v3/users/swift => generated 83 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:25.605237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea4270e-cd35-4171-b924-41f305b1406e None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:25.638580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:25.729110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea4270e-cd35-4171-b924-41f305b1406e None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:25.762749 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe274bf-d4ea-402d-8902-87d2783435ae None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:25.763086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe274bf-d4ea-402d-8902-87d2783435ae None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:25.763299 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe274bf-d4ea-402d-8902-87d2783435ae None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:25.764931 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 266/540] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:25.765542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b133e04e-b938-4ab7-9cc1-4d13e8f2eeb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:25.766384 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b133e04e-b938-4ab7-9cc1-4d13e8f2eeb4 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:25.766863 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b133e04e-b938-4ab7-9cc1-4d13e8f2eeb4 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:25.767329 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b133e04e-b938-4ab7-9cc1-4d13e8f2eeb4 None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:25.768282 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b133e04e-b938-4ab7-9cc1-4d13e8f2eeb4 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:25.768785 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b133e04e-b938-4ab7-9cc1-4d13e8f2eeb4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:25.769706 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482a39a7-ce48-4b76-b377-8e847c643281 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:25.769988 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482a39a7-ce48-4b76-b377-8e847c643281 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:25.770258 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482a39a7-ce48-4b76-b377-8e847c643281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:25.771079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b133e04e-b938-4ab7-9cc1-4d13e8f2eeb4 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:25.790660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 275/541] 10.222.0.36 () {56 vars in 1135 bytes} [Mon Oct 31 14:44:25 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:25.796429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b133e04e-b938-4ab7-9cc1-4d13e8f2eeb4 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:25.801902 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-482a39a7-ce48-4b76-b377-8e847c643281 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:25.803099 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-482a39a7-ce48-4b76-b377-8e847c643281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:25.922481 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 267/542] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:25 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:44:25.933953 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:25.955945 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b133e04e-b938-4ab7-9cc1-4d13e8f2eeb4 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:26.018188 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29707284-5fe2-4374-a98c-7fcb2c27ba0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:26.018829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29707284-5fe2-4374-a98c-7fcb2c27ba0e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:26.019108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29707284-5fe2-4374-a98c-7fcb2c27ba0e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:26.019377 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29707284-5fe2-4374-a98c-7fcb2c27ba0e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:26.039936 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29707284-5fe2-4374-a98c-7fcb2c27ba0e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:26.041467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29707284-5fe2-4374-a98c-7fcb2c27ba0e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:26.064923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-29707284-5fe2-4374-a98c-7fcb2c27ba0e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:26.064923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:26.064923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:26.064923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:26.064923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:26.064923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:26.064923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:26.064923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:26.064923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:26.064923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:26.064923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:26.064923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:26.064923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:26.064923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:26.064923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:26.064923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:26.065930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:26.065930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:26.065930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:26.065930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:26.065930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:26.065930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:26.065930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:26.065930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:26.065930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:26.065930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:26.065930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:26.065930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:26.065930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:26.065930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:26.065930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:26.065930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:26.065930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:26.066802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:26.066802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:26.066802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:26.066802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 276/543] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:44:26.071982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482a39a7-ce48-4b76-b377-8e847c643281 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:26.209956 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482a39a7-ce48-4b76-b377-8e847c643281 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:26.232643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cd0ddc-3919-42b0-8d78-937b8d0c8654 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:26.232946 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cd0ddc-3919-42b0-8d78-937b8d0c8654 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:26.233186 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cd0ddc-3919-42b0-8d78-937b8d0c8654 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:26.234510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 277/544] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:44:26.244084 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80cd0ddc-3919-42b0-8d78-937b8d0c8654 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:26.253795 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:26.267763 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5831511-cb1c-4b92-8744-bd1cbb733cec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:26.268628 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5831511-cb1c-4b92-8744-bd1cbb733cec None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:26.268911 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5831511-cb1c-4b92-8744-bd1cbb733cec None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:26.269187 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5831511-cb1c-4b92-8744-bd1cbb733cec None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:26.270229 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5831511-cb1c-4b92-8744-bd1cbb733cec None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:26.270540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5831511-cb1c-4b92-8744-bd1cbb733cec None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:26.271943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5831511-cb1c-4b92-8744-bd1cbb733cec None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:26.287511 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 268/545] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:26.289109 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a9cebd-1308-4a49-a789-55f224494a48 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:26.289399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a9cebd-1308-4a49-a789-55f224494a48 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:26.289675 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a9cebd-1308-4a49-a789-55f224494a48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:26.314902 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-49a9cebd-1308-4a49-a789-55f224494a48 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:26.316743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-49a9cebd-1308-4a49-a789-55f224494a48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:26.367591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80cd0ddc-3919-42b0-8d78-937b8d0c8654 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:26.400476 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:26.407985 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7431c9fe-094d-46ae-8d4a-9f3f3597610f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:26.408887 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7431c9fe-094d-46ae-8d4a-9f3f3597610f None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:26.409241 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7431c9fe-094d-46ae-8d4a-9f3f3597610f None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:26.409586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7431c9fe-094d-46ae-8d4a-9f3f3597610f None admin] PATH_INFO: `/v3/roles/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:26.419710 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 269/546] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:26 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:44:26.419710 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a9cebd-1308-4a49-a789-55f224494a48 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:26.432598 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7431c9fe-094d-46ae-8d4a-9f3f3597610f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:26.441514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-7431c9fe-094d-46ae-8d4a-9f3f3597610f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:26.441514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:26.441514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:26.441514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:26.441514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:26.441514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:26.441514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:26.441514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:26.441514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:26.441514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:26.441514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:26.441514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:26.441514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:26.441514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:26.441514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:26.441514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:26.442441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:26.442441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:26.442441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:26.442441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:26.442441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:26.442441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:26.442441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:26.442441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:26.442441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:26.442441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:26.442441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:26.442441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:26.442441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:26.442441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:26.442441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:26.442441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:26.443258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:26.443258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:26.443258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:26.443258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:26.443258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:26.443258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:26.443258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:26.443258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:26.443258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:26.443258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:26.443258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:26.443258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:26.452206 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 278/547] 10.222.0.36 () {56 vars in 1102 bytes} [Mon Oct 31 14:44:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:26.455136 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7431c9fe-094d-46ae-8d4a-9f3f3597610f None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:26.571582 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7431c9fe-094d-46ae-8d4a-9f3f3597610f None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:26.608495 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f32672-ea69-4072-b209-85a09a8abb0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:26.609142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f32672-ea69-4072-b209-85a09a8abb0e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:26.609372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f32672-ea69-4072-b209-85a09a8abb0e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:26.609647 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f32672-ea69-4072-b209-85a09a8abb0e None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:26.610428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f32672-ea69-4072-b209-85a09a8abb0e None admin] RBAC: Adding query filter params (name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:26.610730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f32672-ea69-4072-b209-85a09a8abb0e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:26.612317 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f32672-ea69-4072-b209-85a09a8abb0e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:26.622113 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 279/548] 10.222.0.36 () {56 vars in 1103 bytes} [Mon Oct 31 14:44:26 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:26.623302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41053a91-ac2b-4e1d-9ef1-958709bf0e02 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:26.627800 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a9cebd-1308-4a49-a789-55f224494a48 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:26.628172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41053a91-ac2b-4e1d-9ef1-958709bf0e02 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:26.628299 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41053a91-ac2b-4e1d-9ef1-958709bf0e02 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:26.629778 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 280/549] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:26 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:26.630623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41053a91-ac2b-4e1d-9ef1-958709bf0e02 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:26.644542 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:26.666416 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba03839-dd36-4cc8-ac56-5ac3107aa2b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:26.667350 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba03839-dd36-4cc8-ac56-5ac3107aa2b8 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:26.667897 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba03839-dd36-4cc8-ac56-5ac3107aa2b8 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:26.668365 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba03839-dd36-4cc8-ac56-5ac3107aa2b8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:26.675494 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba03839-dd36-4cc8-ac56-5ac3107aa2b8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:26.677236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba03839-dd36-4cc8-ac56-5ac3107aa2b8 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:26.689691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-8ba03839-dd36-4cc8-ac56-5ac3107aa2b8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:26.689691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:26.689691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:26.689691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:26.689691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:26.689691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:26.689691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:26.689691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:26.689691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:26.689691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:26.689691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:26.689691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:26.689691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:26.689691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:26.689691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:26.690614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:26.690614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:26.690614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:26.690614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:26.690614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:26.690614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:26.690614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:26.690614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:26.690614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:26.690614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:26.690614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:26.690614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:26.690614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:26.690614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:26.690614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:26.690614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:26.690614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:26.691490 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:26.691490 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:26.691490 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:26.691490 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:26.691490 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:26.691490 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:26.691490 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:26.691490 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:26.692393 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:26.694630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 270/550] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:26.697761 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bed3eb2-1c7c-4c1f-b907-072df99588cb None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:26.698138 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bed3eb2-1c7c-4c1f-b907-072df99588cb None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:26.698504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bed3eb2-1c7c-4c1f-b907-072df99588cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:26.717580 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-0bed3eb2-1c7c-4c1f-b907-072df99588cb None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:26.719215 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-0bed3eb2-1c7c-4c1f-b907-072df99588cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:26.744296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41053a91-ac2b-4e1d-9ef1-958709bf0e02 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:26.784800 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b36f7e-77cf-40e3-9974-1fee95e6e69e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:26.785479 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b36f7e-77cf-40e3-9974-1fee95e6e69e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:26.785760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b36f7e-77cf-40e3-9974-1fee95e6e69e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:26.786082 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b36f7e-77cf-40e3-9974-1fee95e6e69e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:26.798297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b36f7e-77cf-40e3-9974-1fee95e6e69e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:26.799776 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b36f7e-77cf-40e3-9974-1fee95e6e69e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:26.808159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 271/551] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:26 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 31 14:44:26.809538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bed3eb2-1c7c-4c1f-b907-072df99588cb None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:26.814111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-61b36f7e-77cf-40e3-9974-1fee95e6e69e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:26.814111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:26.814111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:26.814111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:26.814111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:26.814111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:26.814111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:26.814111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:26.814111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:26.814111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:26.814111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:26.814111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:26.814111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:26.814111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:26.814111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:26.814111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:26.815074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:26.815074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:26.815074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:26.815074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:26.815074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:26.815074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:26.815074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:26.815074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:26.815074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:26.815074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:26.815074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:26.815074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:26.815074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:26.815074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:26.815074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:26.815074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:26.815074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:26.815985 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:26.815985 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:26.815985 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:26.815985 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 281/552] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:26.820468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b36f7e-77cf-40e3-9974-1fee95e6e69e None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:26.933309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bed3eb2-1c7c-4c1f-b907-072df99588cb None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:26.935433 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b36f7e-77cf-40e3-9974-1fee95e6e69e None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:26.936508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:26.944326 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:26.978795 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5936b13e-3f4c-4c53-8d86-981e9cc984f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:26.979606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5936b13e-3f4c-4c53-8d86-981e9cc984f5 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:26.980020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5936b13e-3f4c-4c53-8d86-981e9cc984f5 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:26.980870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5936b13e-3f4c-4c53-8d86-981e9cc984f5 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:26.981795 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5936b13e-3f4c-4c53-8d86-981e9cc984f5 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:26.983266 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5936b13e-3f4c-4c53-8d86-981e9cc984f5 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:26.985863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d9468f-bcb1-43b8-bc41-65b519bc124a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:26.986887 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d9468f-bcb1-43b8-bc41-65b519bc124a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:26.987315 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d9468f-bcb1-43b8-bc41-65b519bc124a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:26.987788 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d9468f-bcb1-43b8-bc41-65b519bc124a None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:26.988644 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d9468f-bcb1-43b8-bc41-65b519bc124a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:26.989539 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d9468f-bcb1-43b8-bc41-65b519bc124a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:26.991359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d9468f-bcb1-43b8-bc41-65b519bc124a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:27.000705 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 282/553] 10.222.0.36 () {60 vars in 1141 bytes} [Mon Oct 31 14:44:26 2022] POST /identity/v3/services => generated 240 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:27.001750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5936b13e-3f4c-4c53-8d86-981e9cc984f5 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:27.004898 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 272/554] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:27.011140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d9468f-bcb1-43b8-bc41-65b519bc124a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:27.122576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5936b13e-3f4c-4c53-8d86-981e9cc984f5 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:27.159760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02eb0a05-0448-424c-ba3c-657b334d56ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:27.160497 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02eb0a05-0448-424c-ba3c-657b334d56ba None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:27.160843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02eb0a05-0448-424c-ba3c-657b334d56ba None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:27.161196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02eb0a05-0448-424c-ba3c-657b334d56ba None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:27.162065 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02eb0a05-0448-424c-ba3c-657b334d56ba None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:27.162228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02eb0a05-0448-424c-ba3c-657b334d56ba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:27.163800 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02eb0a05-0448-424c-ba3c-657b334d56ba None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:27.165625 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d9468f-bcb1-43b8-bc41-65b519bc124a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:27.178048 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 283/555] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:27.179109 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02eb0a05-0448-424c-ba3c-657b334d56ba None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:27.207373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0149a857-2867-4cdb-832b-6125ffe4e51a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:27.208460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0149a857-2867-4cdb-832b-6125ffe4e51a None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:27.208923 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0149a857-2867-4cdb-832b-6125ffe4e51a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:27.209451 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0149a857-2867-4cdb-832b-6125ffe4e51a None admin] PATH_INFO: `/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/f4ae5e20cc5b40ccbdb951f908f5080d/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:27.209905 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:27.213182 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0149a857-2867-4cdb-832b-6125ffe4e51a None admin] RBAC: Authorizing `identity:create_grant(project_id=4118e33b099f4cf0a3e3a46b211ba1d8, user_id=f4ae5e20cc5b40ccbdb951f908f5080d, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:27.216799 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0149a857-2867-4cdb-832b-6125ffe4e51a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:27.235393 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 273/556] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:44:27 2022] PUT /identity/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/f4ae5e20cc5b40ccbdb951f908f5080d/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:44:27.236524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0149a857-2867-4cdb-832b-6125ffe4e51a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:27.271203 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:27.314188 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02eb0a05-0448-424c-ba3c-657b334d56ba None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:27.365959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848ad0d8-cf00-4203-b04a-91af83d473fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:27.367088 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848ad0d8-cf00-4203-b04a-91af83d473fa None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:27.367088 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848ad0d8-cf00-4203-b04a-91af83d473fa None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:27.367263 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848ad0d8-cf00-4203-b04a-91af83d473fa None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:27.381434 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848ad0d8-cf00-4203-b04a-91af83d473fa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:27.383746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0149a857-2867-4cdb-832b-6125ffe4e51a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:27.388021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-848ad0d8-cf00-4203-b04a-91af83d473fa None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:44:27.388021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:27.388021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:27.388021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:27.388021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:27.388021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:27.388021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:27.388021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:27.388021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:27.388021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:27.388021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:27.388021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:27.388021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:27.388021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:27.388021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:27.388021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:27.389015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:27.389015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:27.389015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:27.389015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:27.389015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:27.389015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:27.389015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:27.389015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:27.389015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:27.389015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:27.389015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:27.389015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:27.389015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:27.389015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:27.389015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:27.389015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:27.389852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:27.389852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:27.389852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:27.389852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:27.389852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:27.389852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:27.389852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:27.389852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:27.389852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:27.389852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:27.389852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:44:27.389852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:27.390864 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 284/557] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:44:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:27.395519 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848ad0d8-cf00-4203-b04a-91af83d473fa None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:27.435676 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f3a3a3-ed1b-4e77-8016-f3cec91eaec2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:27.436318 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f3a3a3-ed1b-4e77-8016-f3cec91eaec2 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:27.436594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f3a3a3-ed1b-4e77-8016-f3cec91eaec2 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:27.436861 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f3a3a3-ed1b-4e77-8016-f3cec91eaec2 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:27.445658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f3a3a3-ed1b-4e77-8016-f3cec91eaec2 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:27.447118 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f3a3a3-ed1b-4e77-8016-f3cec91eaec2 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:27.460731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-45f3a3a3-ed1b-4e77-8016-f3cec91eaec2 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 31 14:44:27.460731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:27.460731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:27.460731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:27.460731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:27.460731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:27.460731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:27.460731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:27.460731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:27.460731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:27.460731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:27.460731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:27.460731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:27.460731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:27.460731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:27.460731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:27.461978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:27.463141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:27.463141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:27.463141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:27.463141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:27.463141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:27.463141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:27.463141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:27.463141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:27.463141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 31 14:44:27.463141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:27.463141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 274/558] 10.222.0.36 () {56 vars in 1117 bytes} [Mon Oct 31 14:44:27 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:27.470886 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f3a3a3-ed1b-4e77-8016-f3cec91eaec2 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:27.508758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:27.540446 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848ad0d8-cf00-4203-b04a-91af83d473fa None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:27.588834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:27.611569 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f3a3a3-ed1b-4e77-8016-f3cec91eaec2 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:27.620703 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474fbd25-eb12-4979-940f-54dd1f592630 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:27.621462 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474fbd25-eb12-4979-940f-54dd1f592630 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:27.621773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474fbd25-eb12-4979-940f-54dd1f592630 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:27.622161 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474fbd25-eb12-4979-940f-54dd1f592630 None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:27.622834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474fbd25-eb12-4979-940f-54dd1f592630 None admin] RBAC: Adding query filter params (name=admin) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:27.623138 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474fbd25-eb12-4979-940f-54dd1f592630 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:27.624632 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474fbd25-eb12-4979-940f-54dd1f592630 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:27.643348 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 285/559] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:44:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:27.649473 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474fbd25-eb12-4979-940f-54dd1f592630 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:27.651338 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea49294-5290-4781-b060-196b190e5800 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:27.652231 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea49294-5290-4781-b060-196b190e5800 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:27.653369 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea49294-5290-4781-b060-196b190e5800 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:27.653842 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea49294-5290-4781-b060-196b190e5800 None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:27.654768 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea49294-5290-4781-b060-196b190e5800 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:27.655260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea49294-5290-4781-b060-196b190e5800 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:27.656897 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea49294-5290-4781-b060-196b190e5800 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:27.686539 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 275/560] 10.222.0.36 () {56 vars in 1149 bytes} [Mon Oct 31 14:44:27 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:27.703382 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea49294-5290-4781-b060-196b190e5800 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:27.764860 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474fbd25-eb12-4979-940f-54dd1f592630 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:27.799276 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1febe07f-5611-42e8-9bc2-d0896a91a4be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:27.799953 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1febe07f-5611-42e8-9bc2-d0896a91a4be None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:27.800236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1febe07f-5611-42e8-9bc2-d0896a91a4be None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:27.800829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1febe07f-5611-42e8-9bc2-d0896a91a4be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:27.812112 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1febe07f-5611-42e8-9bc2-d0896a91a4be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:27.813682 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1febe07f-5611-42e8-9bc2-d0896a91a4be None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:27.828927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-1febe07f-5611-42e8-9bc2-d0896a91a4be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:27.828927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:27.828927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:27.828927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:27.828927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:27.828927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:27.828927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:27.828927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:27.828927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:27.828927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:27.828927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:27.828927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:27.828927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:27.828927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:27.828927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:27.828927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:27.830076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:27.830076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:27.830076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:27.830076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:27.830076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:27.830076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:27.830076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:27.830076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:27.830076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:27.830076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:27.830076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:27.830076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:27.830076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:27.830076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:27.830076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:27.830076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:27.830076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:27.831098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:27.831098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:27.831098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:27.831700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:27.836106 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 286/561] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:44:27.837771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1febe07f-5611-42e8-9bc2-d0896a91a4be None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:27.940728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea49294-5290-4781-b060-196b190e5800 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:28.010875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f604a08-a9c8-4d30-a5a3-ae2ae66c293e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:28.012773 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f604a08-a9c8-4d30-a5a3-ae2ae66c293e None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:28.013211 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f604a08-a9c8-4d30-a5a3-ae2ae66c293e None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:28.013984 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f604a08-a9c8-4d30-a5a3-ae2ae66c293e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:28.019538 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1febe07f-5611-42e8-9bc2-d0896a91a4be None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:28.027729 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f604a08-a9c8-4d30-a5a3-ae2ae66c293e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:28.030309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f604a08-a9c8-4d30-a5a3-ae2ae66c293e None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:28.032966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:28.052943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-0f604a08-a9c8-4d30-a5a3-ae2ae66c293e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:28.052943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:28.052943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:28.052943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:28.052943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:28.052943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:28.052943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:28.052943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:28.052943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:28.052943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:28.052943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:28.052943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:28.052943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:28.052943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:28.052943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:28.052943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:28.054152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:28.054152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:28.054152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:28.054152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:28.054152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:28.054152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:28.054152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:28.054152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:28.054152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:28.054152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:28.054152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:28.054152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:28.054152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:28.054152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:28.054152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:28.054152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:28.054152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:28.055254 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:28.055254 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:28.055254 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:28.056992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 276/562] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 354 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:44:28.058009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9629b4-0049-4819-ac7c-717d5764006e None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:28.058290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9629b4-0049-4819-ac7c-717d5764006e None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:28.058786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9629b4-0049-4819-ac7c-717d5764006e None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:28.064583 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 277/563] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:28.065845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9629b4-0049-4819-ac7c-717d5764006e None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:28.068583 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a3ea2f-e4cd-450f-9a09-75286aaf7f28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:28.069638 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a3ea2f-e4cd-450f-9a09-75286aaf7f28 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:28.070977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a3ea2f-e4cd-450f-9a09-75286aaf7f28 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:28.071465 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a3ea2f-e4cd-450f-9a09-75286aaf7f28 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:28.072348 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a3ea2f-e4cd-450f-9a09-75286aaf7f28 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:28.072835 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a3ea2f-e4cd-450f-9a09-75286aaf7f28 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:28.075168 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a3ea2f-e4cd-450f-9a09-75286aaf7f28 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:28.093616 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 287/564] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:28.096318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68445b8-75f2-4470-af82-133d49701855 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:28.096866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68445b8-75f2-4470-af82-133d49701855 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:28.097280 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68445b8-75f2-4470-af82-133d49701855 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:28.121622 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-e68445b8-75f2-4470-af82-133d49701855 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:28.122770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e68445b8-75f2-4470-af82-133d49701855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:28.187165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:28.194588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9629b4-0049-4819-ac7c-717d5764006e None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:28.213257 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 288/565] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:28 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:28.214687 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68445b8-75f2-4470-af82-133d49701855 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:28.259480 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d369c2-f1e3-4298-9511-4554a59ecb49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:28.260154 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d369c2-f1e3-4298-9511-4554a59ecb49 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:28.260432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d369c2-f1e3-4298-9511-4554a59ecb49 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:28.260755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d369c2-f1e3-4298-9511-4554a59ecb49 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:28.261418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d369c2-f1e3-4298-9511-4554a59ecb49 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:28.262479 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d369c2-f1e3-4298-9511-4554a59ecb49 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:28.268297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d369c2-f1e3-4298-9511-4554a59ecb49 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:28.290705 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 278/566] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:28.296829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d369c2-f1e3-4298-9511-4554a59ecb49 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:28.343364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68445b8-75f2-4470-af82-133d49701855 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:28.384871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d4f798-e663-4d90-a24e-53a13de26d6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:28.385863 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d4f798-e663-4d90-a24e-53a13de26d6b None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:28.386938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d4f798-e663-4d90-a24e-53a13de26d6b None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:28.387952 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d4f798-e663-4d90-a24e-53a13de26d6b None admin] PATH_INFO: `/v3/users/nova` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:28.399216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d4f798-e663-4d90-a24e-53a13de26d6b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:28.401202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d4f798-e663-4d90-a24e-53a13de26d6b None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:28.409177 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:28.415698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-e2d4f798-e663-4d90-a24e-53a13de26d6b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 31 14:44:28.415698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:28.415698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:28.415698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:28.415698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:28.415698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:28.415698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:28.415698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:28.415698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:28.415698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:28.415698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:28.415698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:28.415698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:28.415698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:28.415698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:28.415698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:28.416685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:28.417660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:28.417660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:28.417660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:28.417660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:28.417660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:28.417660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:28.417660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:28.417660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:28.417660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 31 14:44:28.417660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:28.421556 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 289/567] 10.222.0.36 () {56 vars in 1093 bytes} [Mon Oct 31 14:44:28 2022] GET /identity/v3/users/nova => generated 82 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:28.429363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d4f798-e663-4d90-a24e-53a13de26d6b None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:28.509511 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d369c2-f1e3-4298-9511-4554a59ecb49 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:28.518552 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:28.568251 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d4f798-e663-4d90-a24e-53a13de26d6b None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:28.601236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf942d8-32e0-4966-a3b3-50a8680db5f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:28.602516 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf942d8-32e0-4966-a3b3-50a8680db5f9 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:28.603766 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf942d8-32e0-4966-a3b3-50a8680db5f9 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:28.604653 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf942d8-32e0-4966-a3b3-50a8680db5f9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:28.612852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2c4dc6-3fc5-4289-aebc-2a06151e15d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:28.613593 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2c4dc6-3fc5-4289-aebc-2a06151e15d2 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:28.613966 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2c4dc6-3fc5-4289-aebc-2a06151e15d2 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:28.614413 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2c4dc6-3fc5-4289-aebc-2a06151e15d2 None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:28.615178 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf942d8-32e0-4966-a3b3-50a8680db5f9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:28.615312 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2c4dc6-3fc5-4289-aebc-2a06151e15d2 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:28.615724 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2c4dc6-3fc5-4289-aebc-2a06151e15d2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:28.617176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf942d8-32e0-4966-a3b3-50a8680db5f9 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:28.617322 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2c4dc6-3fc5-4289-aebc-2a06151e15d2 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:28.629090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-bcf942d8-32e0-4966-a3b3-50a8680db5f9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:28.629090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:28.629090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:28.629090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:28.629090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:28.629090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:28.629090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:28.629090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:28.629090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:28.629090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:28.629090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:28.629090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:28.629090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:28.629090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:28.629090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:28.630110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:28.630110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:28.630110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:28.630110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:28.630110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:28.630110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:28.630110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:28.630110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:28.630110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:28.630110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:28.630110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:28.630110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:28.630110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:28.630110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:28.630110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:28.630110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:28.630110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:28.631009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:28.631009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:28.631009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:28.631009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:28.631009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:28.631009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:28.631009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:28.631009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:28.631009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 279/568] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 334 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:44:28.631517 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf942d8-32e0-4966-a3b3-50a8680db5f9 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:28.640054 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 290/569] 10.222.0.36 () {56 vars in 1133 bytes} [Mon Oct 31 14:44:28 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:28.640556 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2c4dc6-3fc5-4289-aebc-2a06151e15d2 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:28.751618 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2c4dc6-3fc5-4289-aebc-2a06151e15d2 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:28.754087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:28.758065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf942d8-32e0-4966-a3b3-50a8680db5f9 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:28.774168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:28.792522 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18dcc51-029e-4dae-ada3-291e7a176e66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:28.793490 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18dcc51-029e-4dae-ada3-291e7a176e66 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:28.793770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18dcc51-029e-4dae-ada3-291e7a176e66 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:28.794039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18dcc51-029e-4dae-ada3-291e7a176e66 None admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:28.794752 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18dcc51-029e-4dae-ada3-291e7a176e66 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:28.795176 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18dcc51-029e-4dae-ada3-291e7a176e66 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:28.796774 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18dcc51-029e-4dae-ada3-291e7a176e66 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:28.801042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dff5bc5-6681-466f-a6e6-fd58fda66cd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:28.801904 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dff5bc5-6681-466f-a6e6-fd58fda66cd8 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:28.802405 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dff5bc5-6681-466f-a6e6-fd58fda66cd8 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:28.802879 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dff5bc5-6681-466f-a6e6-fd58fda66cd8 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:28.804290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dff5bc5-6681-466f-a6e6-fd58fda66cd8 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:28.806447 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dff5bc5-6681-466f-a6e6-fd58fda66cd8 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:28.810190 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 291/570] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:28.811094 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18dcc51-029e-4dae-ada3-291e7a176e66 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:28.818394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-1dff5bc5-6681-466f-a6e6-fd58fda66cd8 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 31 14:44:28.818394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:28.818394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:28.818394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:28.818394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:28.818394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:28.818394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:28.818394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:28.818394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 31 14:44:28.818394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:28.818394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:28.818394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:28.818394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:28.818394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:28.818394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:28.818394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:28.818394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:28.819457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:28.819457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:28.819457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:28.819457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:28.819457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:28.819457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:28.819457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:28.819457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:28.819457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:28.819457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:28.819457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:28.819457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:28.819457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:28.819457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:28.819457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:28.819457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:28.820395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:28.820395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:28.820395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:28.820395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:28.820395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:28.820395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:28.820395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:28.820395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:28.820395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:28.820395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:28.820395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:28.820395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:28.820395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 31 14:44:28.820395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:28.822975 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 280/571] 10.222.0.36 () {56 vars in 1129 bytes} [Mon Oct 31 14:44:28 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:28.825058 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dff5bc5-6681-466f-a6e6-fd58fda66cd8 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:28.934339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18dcc51-029e-4dae-ada3-291e7a176e66 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:28.954194 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dff5bc5-6681-466f-a6e6-fd58fda66cd8 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:28.975787 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b9f79e-29cf-4e27-9efa-be11561b5ef0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:28.976664 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b9f79e-29cf-4e27-9efa-be11561b5ef0 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:28.977127 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b9f79e-29cf-4e27-9efa-be11561b5ef0 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:28.977489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b9f79e-29cf-4e27-9efa-be11561b5ef0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:28.990660 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c35496-23e5-4d34-87a5-9e1c784417ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:28.991416 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c35496-23e5-4d34-87a5-9e1c784417ce None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:28.991744 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c35496-23e5-4d34-87a5-9e1c784417ce None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:28.992019 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c35496-23e5-4d34-87a5-9e1c784417ce None admin] PATH_INFO: `/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/e0d0f8598fac40449bcb2a55ff8df8d9/roles/88d62c66100a4d04a45e3607135bdd0c` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:29.008392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c35496-23e5-4d34-87a5-9e1c784417ce None admin] RBAC: Authorizing `identity:create_grant(project_id=4118e33b099f4cf0a3e3a46b211ba1d8, user_id=e0d0f8598fac40449bcb2a55ff8df8d9, role_id=88d62c66100a4d04a45e3607135bdd0c)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:29.010592 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c35496-23e5-4d34-87a5-9e1c784417ce None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:29.012894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:29.028776 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 281/572] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:44:28 2022] PUT /identity/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/e0d0f8598fac40449bcb2a55ff8df8d9/roles/88d62c66100a4d04a45e3607135bdd0c => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:44:29.029685 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c35496-23e5-4d34-87a5-9e1c784417ce None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:29.064495 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b9f79e-29cf-4e27-9efa-be11561b5ef0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:29.066723 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b9f79e-29cf-4e27-9efa-be11561b5ef0 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:29.069562 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:29.155375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c35496-23e5-4d34-87a5-9e1c784417ce None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:29.195309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec0fb08-415c-4511-b37c-0a9b379dea65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:29.196104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec0fb08-415c-4511-b37c-0a9b379dea65 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:29.196263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec0fb08-415c-4511-b37c-0a9b379dea65 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:29.196453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec0fb08-415c-4511-b37c-0a9b379dea65 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:29.197093 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec0fb08-415c-4511-b37c-0a9b379dea65 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:29.197407 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 292/573] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:44:28 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 31 14:44:29.197699 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec0fb08-415c-4511-b37c-0a9b379dea65 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:29.199335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec0fb08-415c-4511-b37c-0a9b379dea65 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:29.204273 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b9f79e-29cf-4e27-9efa-be11561b5ef0 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:29.206347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 282/574] 10.222.0.36 () {56 vars in 1124 bytes} [Mon Oct 31 14:44:29 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:29.212332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec0fb08-415c-4511-b37c-0a9b379dea65 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:29.319945 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:29.368666 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b9f79e-29cf-4e27-9efa-be11561b5ef0 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:29.371957 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:29.382603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec0fb08-415c-4511-b37c-0a9b379dea65 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:29.407403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e319e-8fce-4999-8a9c-dadd1c26c440 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:29.408098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e319e-8fce-4999-8a9c-dadd1c26c440 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:29.408374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e319e-8fce-4999-8a9c-dadd1c26c440 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:29.408644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e319e-8fce-4999-8a9c-dadd1c26c440 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:29.419578 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5e319e-8fce-4999-8a9c-dadd1c26c440 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:29.421247 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5e319e-8fce-4999-8a9c-dadd1c26c440 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:29.421772 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b705d53e-42c5-49f4-8f24-453cffad6304 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:29.422396 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b705d53e-42c5-49f4-8f24-453cffad6304 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:29.422676 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b705d53e-42c5-49f4-8f24-453cffad6304 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:29.423054 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b705d53e-42c5-49f4-8f24-453cffad6304 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:29.423843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b705d53e-42c5-49f4-8f24-453cffad6304 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:29.424159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b705d53e-42c5-49f4-8f24-453cffad6304 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:29.425578 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b705d53e-42c5-49f4-8f24-453cffad6304 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:29.431803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 283/575] 10.222.0.36 () {56 vars in 1124 bytes} [Mon Oct 31 14:44:29 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:29.440352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-ab5e319e-8fce-4999-8a9c-dadd1c26c440 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:29.440352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:29.440352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:29.440352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:29.440352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:29.440352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:29.440352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:29.440352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:29.440352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:29.440352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:29.440352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:29.440352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:29.440352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:29.440352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:29.440352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:29.440352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:29.442506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:29.442506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:29.442506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:29.442506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:29.442506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:29.442506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:29.442506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:29.442506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:29.442506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:29.442506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:29.442506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:29.442506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:29.442506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:29.442506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:29.442506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:29.442506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:29.442506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:29.444931 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:29.444931 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:29.444931 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:29.444931 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 293/576] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:29.445444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b705d53e-42c5-49f4-8f24-453cffad6304 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:29.448828 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e319e-8fce-4999-8a9c-dadd1c26c440 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:29.648435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b705d53e-42c5-49f4-8f24-453cffad6304 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:29.663678 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e319e-8fce-4999-8a9c-dadd1c26c440 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:29.702083 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:29.725340 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a4bccc-07d6-406a-b791-4a22c18192ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:29.725956 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a4bccc-07d6-406a-b791-4a22c18192ff None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:29.726236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a4bccc-07d6-406a-b791-4a22c18192ff None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:29.726508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a4bccc-07d6-406a-b791-4a22c18192ff None admin] PATH_INFO: `/v3/endpoints` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:29.727490 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a4bccc-07d6-406a-b791-4a22c18192ff None admin] RBAC: Adding query filter params (interface=public, service_id=7b0c9aa758534fffbd4b25b823e98555, region_id=RegionOne) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:29.727956 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a4bccc-07d6-406a-b791-4a22c18192ff None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:29.729357 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a4bccc-07d6-406a-b791-4a22c18192ff None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:29.739414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 284/577] 10.222.0.36 () {56 vars in 1251 bytes} [Mon Oct 31 14:44:29 2022] GET /identity/v3/endpoints?service_id=7b0c9aa758534fffbd4b25b823e98555&interface=public®ion_id=RegionOne => generated 194 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:29.740419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9c8587-9a10-43e3-a2e2-62b3c59a3d24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:29.741052 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9c8587-9a10-43e3-a2e2-62b3c59a3d24 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:29.741332 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9c8587-9a10-43e3-a2e2-62b3c59a3d24 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:29.741606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9c8587-9a10-43e3-a2e2-62b3c59a3d24 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:29.742257 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9c8587-9a10-43e3-a2e2-62b3c59a3d24 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:29.742560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9c8587-9a10-43e3-a2e2-62b3c59a3d24 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:29.744060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9c8587-9a10-43e3-a2e2-62b3c59a3d24 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:29.744764 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a4bccc-07d6-406a-b791-4a22c18192ff None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:29.769855 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:29.770162 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 294/578] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:29.775068 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1bacd3-b036-4d09-8824-54a7849e8ae4 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:29.775068 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1bacd3-b036-4d09-8824-54a7849e8ae4 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:29.775068 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1bacd3-b036-4d09-8824-54a7849e8ae4 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:29.775068 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 295/579] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:29.779195 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1bacd3-b036-4d09-8824-54a7849e8ae4 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:29.860047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a4bccc-07d6-406a-b791-4a22c18192ff None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:29.898734 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1bacd3-b036-4d09-8824-54a7849e8ae4 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:29.900415 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb683561-2652-499e-aba5-5d2fe6370601 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:29.901003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb683561-2652-499e-aba5-5d2fe6370601 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:29.901287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb683561-2652-499e-aba5-5d2fe6370601 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:29.901565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb683561-2652-499e-aba5-5d2fe6370601 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:29.902909 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb683561-2652-499e-aba5-5d2fe6370601 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:29.904323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb683561-2652-499e-aba5-5d2fe6370601 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:29.909426 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 285/580] 10.222.0.36 () {56 vars in 1087 bytes} [Mon Oct 31 14:44:29 2022] GET /identity/v3/services => generated 772 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:29.910607 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206802b3-4040-47f1-9b99-ba34f7edbeb2 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:29.910891 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206802b3-4040-47f1-9b99-ba34f7edbeb2 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:29.911165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206802b3-4040-47f1-9b99-ba34f7edbeb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:29.931002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-206802b3-4040-47f1-9b99-ba34f7edbeb2 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:29.932275 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-206802b3-4040-47f1-9b99-ba34f7edbeb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:29.955151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d8b389-c1d2-4843-91d6-c42e50dd6044 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:29.956024 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d8b389-c1d2-4843-91d6-c42e50dd6044 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:29.956310 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d8b389-c1d2-4843-91d6-c42e50dd6044 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:29.956632 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d8b389-c1d2-4843-91d6-c42e50dd6044 None admin] PATH_INFO: `/v3/projects/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:29.964709 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d8b389-c1d2-4843-91d6-c42e50dd6044 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:29.966709 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d8b389-c1d2-4843-91d6-c42e50dd6044 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:29.982560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-b3d8b389-c1d2-4843-91d6-c42e50dd6044 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:29.982560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:29.982560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:29.982560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:29.982560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:29.982560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:29.982560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:29.982560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:29.982560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:29.982560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:29.982560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:29.982560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:29.982560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:29.982560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:29.982560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:29.983609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:29.983609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:29.983609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:29.983609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:29.983609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:29.983609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:29.983609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:29.983609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:29.983609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:29.983609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:29.983609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:29.983609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:29.983609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:29.983609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:29.983609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:29.983609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:29.983609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:29.989970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:29.989970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:29.989970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:29.989970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:29.989970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:29.989970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:29.989970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:29.989970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:29.989970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 296/581] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:29.989970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbfd71f-5969-4e7d-9eb3-c408ba0e5eba None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:29.989970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbfd71f-5969-4e7d-9eb3-c408ba0e5eba None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:29.990594 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbfd71f-5969-4e7d-9eb3-c408ba0e5eba None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:29.990594 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 297/582] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:29.990594 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dbfd71f-5969-4e7d-9eb3-c408ba0e5eba None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:30.032000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 286/583] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:29 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:30.033909 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de225160-af8e-4adb-87bd-f8a76e8ca37a None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:30.034424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de225160-af8e-4adb-87bd-f8a76e8ca37a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:30.034905 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de225160-af8e-4adb-87bd-f8a76e8ca37a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:30.054824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:30.059268 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-de225160-af8e-4adb-87bd-f8a76e8ca37a None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:30.060845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-de225160-af8e-4adb-87bd-f8a76e8ca37a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:30.076995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:30.154575 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 287/584] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:30 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:30.168956 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dbfd71f-5969-4e7d-9eb3-c408ba0e5eba None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:30.212252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a58f653-f1b8-4ab7-bde2-f65cd6f84911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:30.213182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a58f653-f1b8-4ab7-bde2-f65cd6f84911 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:30.213650 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a58f653-f1b8-4ab7-bde2-f65cd6f84911 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:30.214152 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a58f653-f1b8-4ab7-bde2-f65cd6f84911 None admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:30.215724 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a58f653-f1b8-4ab7-bde2-f65cd6f84911 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:30.216230 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a58f653-f1b8-4ab7-bde2-f65cd6f84911 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:30.217830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a58f653-f1b8-4ab7-bde2-f65cd6f84911 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:30.230724 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 298/585] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:30.236228 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de225160-af8e-4adb-87bd-f8a76e8ca37a None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:30.352839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a58f653-f1b8-4ab7-bde2-f65cd6f84911 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:30.356525 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de225160-af8e-4adb-87bd-f8a76e8ca37a None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:30.399087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ed1ee9-af5e-4ea9-94d5-084eac11eb94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:30.399717 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ed1ee9-af5e-4ea9-94d5-084eac11eb94 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:30.399994 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ed1ee9-af5e-4ea9-94d5-084eac11eb94 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:30.400262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ed1ee9-af5e-4ea9-94d5-084eac11eb94 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:30.400999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ed1ee9-af5e-4ea9-94d5-084eac11eb94 None admin] RBAC: Adding query filter params (role.id=57572a37675d4c418e13794a7b8f3a5d, scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8, user.id=c8cace029a92406f850bb0a4ca07c461) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:30.401314 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ed1ee9-af5e-4ea9-94d5-084eac11eb94 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:30.403582 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ed1ee9-af5e-4ea9-94d5-084eac11eb94 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:30.417284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 288/586] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:44:30 2022] GET /identity/v3/role_assignments?user.id=c8cace029a92406f850bb0a4ca07c461&scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8&role.id=57572a37675d4c418e13794a7b8f3a5d => generated 608 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:30.465167 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ed1ee9-af5e-4ea9-94d5-084eac11eb94 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:30.479935 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:30.516141 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:30.523260 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a58f653-f1b8-4ab7-bde2-f65cd6f84911 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:30.573314 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13edd4b3-1588-4ccf-adcb-21ba4edc0a70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:30.574663 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13edd4b3-1588-4ccf-adcb-21ba4edc0a70 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:30.575033 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13edd4b3-1588-4ccf-adcb-21ba4edc0a70 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:30.575398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13edd4b3-1588-4ccf-adcb-21ba4edc0a70 None admin] PATH_INFO: `/v3/services/network` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:30.576114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13edd4b3-1588-4ccf-adcb-21ba4edc0a70 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:30.578135 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13edd4b3-1588-4ccf-adcb-21ba4edc0a70 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:30.590768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-13edd4b3-1588-4ccf-adcb-21ba4edc0a70 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 31 14:44:30.590768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:30.590768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:30.590768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:30.590768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:30.590768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:30.590768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:30.590768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:30.590768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 31 14:44:30.590768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:30.590768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:30.590768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:30.590768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:30.590768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:30.590768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:30.590768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:30.590768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:30.591915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:30.591915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:30.591915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:30.591915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:30.591915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:30.591915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:30.591915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:30.591915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:30.591915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:30.591915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:30.591915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:30.591915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:30.591915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:30.591915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:30.591915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:30.591915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:30.593079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:30.593079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:30.593079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:30.593079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:30.593079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:30.593079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:30.593079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:30.593079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:30.593079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:30.593079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:30.593079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:30.593079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:30.593079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 31 14:44:30.593079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:30.593079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 299/587] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:30 2022] GET /identity/v3/services/network => generated 88 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:44:30.598727 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13edd4b3-1588-4ccf-adcb-21ba4edc0a70 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:30.688759 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ed1ee9-af5e-4ea9-94d5-084eac11eb94 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:30.727403 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91f35a8-3c24-4ea8-8c14-a88576605a58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:30.728191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91f35a8-3c24-4ea8-8c14-a88576605a58 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:30.728530 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91f35a8-3c24-4ea8-8c14-a88576605a58 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:30.728906 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91f35a8-3c24-4ea8-8c14-a88576605a58 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:30.732089 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13edd4b3-1588-4ccf-adcb-21ba4edc0a70 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:30.740432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91f35a8-3c24-4ea8-8c14-a88576605a58 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:30.745969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-c91f35a8-3c24-4ea8-8c14-a88576605a58 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:44:30.745969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:30.745969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:30.745969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:30.745969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:30.745969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:30.745969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:30.745969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:30.745969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:30.745969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:30.745969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:30.745969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:30.745969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:30.745969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:30.745969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:30.745969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:30.747865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:30.747865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:30.747865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:30.747865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:30.747865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:30.747865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:30.747865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:30.747865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:30.747865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:30.747865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:30.747865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:30.747865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:30.747865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:30.747865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:30.747865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:30.747865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:30.750048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:30.750048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:30.750048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:30.750048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:30.750048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:30.750048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:30.750048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:30.750048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:30.750048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:30.750048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:30.750048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:44:30.750048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:30.750048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 289/588] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:44:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:44:30.761021 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91f35a8-3c24-4ea8-8c14-a88576605a58 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:30.772603 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c88b2c-dfad-4ccd-9b67-f13b2d173336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:30.773217 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c88b2c-dfad-4ccd-9b67-f13b2d173336 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:30.774040 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c88b2c-dfad-4ccd-9b67-f13b2d173336 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:30.774324 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c88b2c-dfad-4ccd-9b67-f13b2d173336 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:30.774957 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c88b2c-dfad-4ccd-9b67-f13b2d173336 None admin] RBAC: Adding query filter params (name=network) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:30.775265 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c88b2c-dfad-4ccd-9b67-f13b2d173336 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:30.777639 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c88b2c-dfad-4ccd-9b67-f13b2d173336 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:30.783510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 300/589] 10.222.0.36 () {56 vars in 1112 bytes} [Mon Oct 31 14:44:30 2022] GET /identity/v3/services?name=network => generated 124 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:30.789422 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c88b2c-dfad-4ccd-9b67-f13b2d173336 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:30.818418 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:30.842192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:30.881320 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91f35a8-3c24-4ea8-8c14-a88576605a58 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:30.908826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c88b2c-dfad-4ccd-9b67-f13b2d173336 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:30.917240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8aaeab-dbba-4121-a083-7dfabcca5bb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:30.917959 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8aaeab-dbba-4121-a083-7dfabcca5bb9 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:30.918241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8aaeab-dbba-4121-a083-7dfabcca5bb9 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:30.918618 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8aaeab-dbba-4121-a083-7dfabcca5bb9 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:30.919294 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8aaeab-dbba-4121-a083-7dfabcca5bb9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:30.919628 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8aaeab-dbba-4121-a083-7dfabcca5bb9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:30.921067 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8aaeab-dbba-4121-a083-7dfabcca5bb9 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:30.930602 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 290/590] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:44:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:30.938121 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8aaeab-dbba-4121-a083-7dfabcca5bb9 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:30.955275 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0a322a-12ef-4029-b57e-d21ed450a969 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:30.955849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0a322a-12ef-4029-b57e-d21ed450a969 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:30.956130 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0a322a-12ef-4029-b57e-d21ed450a969 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:30.956408 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0a322a-12ef-4029-b57e-d21ed450a969 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:30.957141 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0a322a-12ef-4029-b57e-d21ed450a969 None admin] RBAC: Adding query filter params (type=network) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:30.957448 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0a322a-12ef-4029-b57e-d21ed450a969 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:30.958992 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0a322a-12ef-4029-b57e-d21ed450a969 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:30.965538 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 301/591] 10.222.0.36 () {56 vars in 1112 bytes} [Mon Oct 31 14:44:30 2022] GET /identity/v3/services?type=network => generated 350 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:30.973552 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0a322a-12ef-4029-b57e-d21ed450a969 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:31.110124 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8aaeab-dbba-4121-a083-7dfabcca5bb9 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:31.153149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:31.158540 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:31.165249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87902dcc-632f-4b26-b754-5e8fe30d24ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:31.165896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87902dcc-632f-4b26-b754-5e8fe30d24ba None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:31.166173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87902dcc-632f-4b26-b754-5e8fe30d24ba None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:31.166443 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87902dcc-632f-4b26-b754-5e8fe30d24ba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:31.188298 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0a322a-12ef-4029-b57e-d21ed450a969 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:31.191411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87902dcc-632f-4b26-b754-5e8fe30d24ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:31.192907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87902dcc-632f-4b26-b754-5e8fe30d24ba None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:31.218577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-87902dcc-632f-4b26-b754-5e8fe30d24ba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:31.218577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:31.218577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:31.218577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:31.218577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:31.218577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:31.218577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:31.218577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:31.218577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:31.218577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:31.218577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:31.218577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:31.218577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:31.218577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:31.218577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:31.218577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:31.219841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:31.219841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:31.219841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:31.219841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:31.219841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:31.219841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:31.219841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:31.219841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:31.219841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:31.219841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:31.219841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:31.219841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:31.219841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:31.219841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:31.219841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:31.219841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:31.219841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:31.220766 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:31.220766 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:31.220766 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:31.220766 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 291/592] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:31.225823 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87902dcc-632f-4b26-b754-5e8fe30d24ba None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:31.265215 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b27943c-721d-41e5-ac87-cc3fd9498eb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:31.265830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b27943c-721d-41e5-ac87-cc3fd9498eb8 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:31.266105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b27943c-721d-41e5-ac87-cc3fd9498eb8 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:31.266372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b27943c-721d-41e5-ac87-cc3fd9498eb8 None admin] PATH_INFO: `/v3/endpoints` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:31.267065 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b27943c-721d-41e5-ac87-cc3fd9498eb8 None admin] RBAC: Adding query filter params (interface=public, service_id=1827048dc96a4a549dde503b7d671f90, region_id=RegionOne) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:31.267366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b27943c-721d-41e5-ac87-cc3fd9498eb8 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:31.268948 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b27943c-721d-41e5-ac87-cc3fd9498eb8 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:31.277777 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 302/593] 10.222.0.36 () {56 vars in 1251 bytes} [Mon Oct 31 14:44:30 2022] GET /identity/v3/endpoints?service_id=1827048dc96a4a549dde503b7d671f90&interface=public®ion_id=RegionOne => generated 194 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:31.283384 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b27943c-721d-41e5-ac87-cc3fd9498eb8 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:31.393410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b27943c-721d-41e5-ac87-cc3fd9498eb8 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:31.429449 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af264c9-48e1-4b31-adfa-7779ebfc9184 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:31.430198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af264c9-48e1-4b31-adfa-7779ebfc9184 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:31.430476 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af264c9-48e1-4b31-adfa-7779ebfc9184 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:31.430741 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af264c9-48e1-4b31-adfa-7779ebfc9184 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:31.431337 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af264c9-48e1-4b31-adfa-7779ebfc9184 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:31.432729 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af264c9-48e1-4b31-adfa-7779ebfc9184 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:31.438656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 303/594] 10.222.0.36 () {56 vars in 1087 bytes} [Mon Oct 31 14:44:31 2022] GET /identity/v3/services => generated 772 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:31.474788 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87902dcc-632f-4b26-b754-5e8fe30d24ba None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:31.546773 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92874749-4f68-4949-929d-047d45f3f052 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:31.547408 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92874749-4f68-4949-929d-047d45f3f052 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:31.547716 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92874749-4f68-4949-929d-047d45f3f052 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:31.547993 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92874749-4f68-4949-929d-047d45f3f052 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:31.548656 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92874749-4f68-4949-929d-047d45f3f052 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:31.548959 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92874749-4f68-4949-929d-047d45f3f052 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:31.550369 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92874749-4f68-4949-929d-047d45f3f052 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:31.573711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 292/595] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 345 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:31.580727 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af264c9-48e1-4b31-adfa-7779ebfc9184 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:31.620311 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:31.719824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af264c9-48e1-4b31-adfa-7779ebfc9184 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:31.723027 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5c6e91-5baf-4174-b006-0b6746bad9f3 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:31.723315 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5c6e91-5baf-4174-b006-0b6746bad9f3 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:31.723698 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5c6e91-5baf-4174-b006-0b6746bad9f3 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:31.727923 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 293/596] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:44:31.735937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531390b8-4cb4-41fd-a0df-2adac3521525 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:31.736227 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531390b8-4cb4-41fd-a0df-2adac3521525 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:31.736490 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531390b8-4cb4-41fd-a0df-2adac3521525 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:31.760796 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-531390b8-4cb4-41fd-a0df-2adac3521525 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:31.763379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-531390b8-4cb4-41fd-a0df-2adac3521525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:31.767357 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9dcab1-13ae-4062-a558-02455087da63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:31.767868 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9dcab1-13ae-4062-a558-02455087da63 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:31.768154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9dcab1-13ae-4062-a558-02455087da63 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:31.769162 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9dcab1-13ae-4062-a558-02455087da63 None admin] PATH_INFO: `/v3/users/swift` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:31.781942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9dcab1-13ae-4062-a558-02455087da63 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:31.783756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9dcab1-13ae-4062-a558-02455087da63 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:31.797824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-fe9dcab1-13ae-4062-a558-02455087da63 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 31 14:44:31.797824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:31.797824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:31.797824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:31.797824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:31.797824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:31.797824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:31.797824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:31.797824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:31.797824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:31.797824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:31.797824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:31.797824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:31.797824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:31.797824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:31.797824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:31.798771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:31.799759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:31.799759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:31.799759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:31.799759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:31.799759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:31.799759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:31.799759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:31.799759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:31.799759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 31 14:44:31.799759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:31.801129 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 304/597] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:44:31 2022] GET /identity/v3/users/swift => generated 83 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:31.807644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9dcab1-13ae-4062-a558-02455087da63 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:31.821744 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:31.862187 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 294/598] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:31 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:31.901669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:31.935460 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9dcab1-13ae-4062-a558-02455087da63 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:31.979593 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4a2f5a-505b-4dfe-a656-b984d9696acc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:31.980688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4a2f5a-505b-4dfe-a656-b984d9696acc None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:31.980979 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4a2f5a-505b-4dfe-a656-b984d9696acc None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:31.981433 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4a2f5a-505b-4dfe-a656-b984d9696acc None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:31.982175 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4a2f5a-505b-4dfe-a656-b984d9696acc None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:31.982488 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4a2f5a-505b-4dfe-a656-b984d9696acc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:31.983944 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4a2f5a-505b-4dfe-a656-b984d9696acc None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:32.002740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 305/599] 10.222.0.36 () {56 vars in 1135 bytes} [Mon Oct 31 14:44:31 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:32.008086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531390b8-4cb4-41fd-a0df-2adac3521525 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:32.164603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531390b8-4cb4-41fd-a0df-2adac3521525 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:32.220352 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c982ce11-ea3d-4a5a-bda7-de07dd26fe5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:32.220986 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c982ce11-ea3d-4a5a-bda7-de07dd26fe5d None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:32.221321 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c982ce11-ea3d-4a5a-bda7-de07dd26fe5d None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:32.221601 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c982ce11-ea3d-4a5a-bda7-de07dd26fe5d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:32.270966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:32.286131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4a2f5a-505b-4dfe-a656-b984d9696acc None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:32.309887 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c982ce11-ea3d-4a5a-bda7-de07dd26fe5d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:32.311354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c982ce11-ea3d-4a5a-bda7-de07dd26fe5d None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:32.482167 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4a2f5a-505b-4dfe-a656-b984d9696acc None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:32.496172 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 295/600] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:44:32 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 489 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 31 14:44:32.501696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c982ce11-ea3d-4a5a-bda7-de07dd26fe5d None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:32.552761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb31913-9328-416e-856a-b4e769235b4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:32.553390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb31913-9328-416e-856a-b4e769235b4e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:32.553667 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb31913-9328-416e-856a-b4e769235b4e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:32.553935 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb31913-9328-416e-856a-b4e769235b4e None admin] PATH_INFO: `/v3/roles/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:32.558834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:32.579650 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb31913-9328-416e-856a-b4e769235b4e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:32.584443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-bbb31913-9328-416e-856a-b4e769235b4e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:32.584443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:32.584443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:32.584443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:32.584443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:32.584443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:32.584443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:32.584443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:32.584443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:32.584443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:32.584443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:32.584443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:32.584443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:32.584443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:32.584443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:32.584443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:32.585376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:32.585376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:32.585376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:32.585376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:32.585376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:32.585376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:32.585376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:32.585376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:32.585376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:32.585376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:32.585376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:32.585376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:32.585376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:32.585376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:32.585376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:32.585376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:32.586218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:32.586218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:32.586218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:32.586218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:32.586218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:32.586218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:32.586218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:32.586218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:32.586218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:32.586218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:32.586218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:44:32.586218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:32.589409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 306/601] 10.222.0.36 () {56 vars in 1102 bytes} [Mon Oct 31 14:44:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:32.591293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb31913-9328-416e-856a-b4e769235b4e None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:32.643973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:32.710654 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c982ce11-ea3d-4a5a-bda7-de07dd26fe5d None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:32.763797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c664d1-4b41-4fa6-8771-baaa3e0cc0b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:32.764543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c664d1-4b41-4fa6-8771-baaa3e0cc0b4 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:32.764837 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c664d1-4b41-4fa6-8771-baaa3e0cc0b4 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:32.765107 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c664d1-4b41-4fa6-8771-baaa3e0cc0b4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:32.773796 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb31913-9328-416e-856a-b4e769235b4e None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:32.776631 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c664d1-4b41-4fa6-8771-baaa3e0cc0b4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:32.778744 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c664d1-4b41-4fa6-8771-baaa3e0cc0b4 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:32.795867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-67c664d1-4b41-4fa6-8771-baaa3e0cc0b4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:32.795867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:32.795867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:32.795867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:32.795867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:32.795867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:32.795867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:32.795867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:32.795867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:32.795867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:32.795867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:32.795867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:32.795867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:32.795867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:32.795867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:32.795867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:32.796821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:32.796821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:32.796821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:32.796821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:32.796821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:32.796821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:32.796821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:32.796821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:32.796821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:32.796821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:32.796821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:32.796821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:32.796821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:32.796821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:32.796821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:32.796821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:32.796821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:32.797736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:32.797736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:32.797736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:32.800028 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 296/602] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:32.801511 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd070140-af8e-44ad-b26f-90715f45f624 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:32.802086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd070140-af8e-44ad-b26f-90715f45f624 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:32.802573 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd070140-af8e-44ad-b26f-90715f45f624 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:32.808910 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 297/603] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:32 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:32.809984 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd070140-af8e-44ad-b26f-90715f45f624 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:32.811177 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2258fd99-9a26-4a1c-a7f9-7476a87bdf40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:32.811902 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2258fd99-9a26-4a1c-a7f9-7476a87bdf40 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:32.812250 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2258fd99-9a26-4a1c-a7f9-7476a87bdf40 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:32.812591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2258fd99-9a26-4a1c-a7f9-7476a87bdf40 None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:32.813339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2258fd99-9a26-4a1c-a7f9-7476a87bdf40 None admin] RBAC: Adding query filter params (name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:32.813711 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2258fd99-9a26-4a1c-a7f9-7476a87bdf40 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:32.816462 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2258fd99-9a26-4a1c-a7f9-7476a87bdf40 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:32.831189 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 307/604] 10.222.0.36 () {56 vars in 1103 bytes} [Mon Oct 31 14:44:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:32.831189 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74396656-6899-4336-b065-2373bee8db3c None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:32.831189 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74396656-6899-4336-b065-2373bee8db3c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:32.831189 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74396656-6899-4336-b065-2373bee8db3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:32.847779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-74396656-6899-4336-b065-2373bee8db3c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:32.849503 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-74396656-6899-4336-b065-2373bee8db3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:32.912428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:32.928958 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd070140-af8e-44ad-b26f-90715f45f624 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:32.937271 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 308/605] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:32 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:32.943593 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74396656-6899-4336-b065-2373bee8db3c None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:32.981791 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:33.010574 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd28d38-9065-4183-b4f7-c2112f9da0c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:33.011761 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd28d38-9065-4183-b4f7-c2112f9da0c5 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:33.012272 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd28d38-9065-4183-b4f7-c2112f9da0c5 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:33.012733 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd28d38-9065-4183-b4f7-c2112f9da0c5 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:33.013565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd28d38-9065-4183-b4f7-c2112f9da0c5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:33.014703 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd28d38-9065-4183-b4f7-c2112f9da0c5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:33.017239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd28d38-9065-4183-b4f7-c2112f9da0c5 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:33.031625 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 298/606] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:33.036675 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd28d38-9065-4183-b4f7-c2112f9da0c5 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:33.096863 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74396656-6899-4336-b065-2373bee8db3c None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:33.140974 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7221d71-4b7b-4a86-9c68-b3b0715fa5da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:33.141623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7221d71-4b7b-4a86-9c68-b3b0715fa5da None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:33.141902 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7221d71-4b7b-4a86-9c68-b3b0715fa5da None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:33.142175 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7221d71-4b7b-4a86-9c68-b3b0715fa5da None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:33.154979 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7221d71-4b7b-4a86-9c68-b3b0715fa5da None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:33.156461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7221d71-4b7b-4a86-9c68-b3b0715fa5da None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:33.174730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-b7221d71-4b7b-4a86-9c68-b3b0715fa5da None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:33.174730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:33.174730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:33.174730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:33.174730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:33.174730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:33.174730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:33.174730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:33.174730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:33.174730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:33.174730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:33.174730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:33.174730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:33.174730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:33.174730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:33.174730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:33.175744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:33.175744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:33.175744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:33.175744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:33.175744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:33.175744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:33.175744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:33.175744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:33.175744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:33.175744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:33.175744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:33.175744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:33.175744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:33.175744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:33.175744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:33.175744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:33.175744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:33.176633 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:33.176633 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:33.176633 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:33.176978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 309/607] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:33.178544 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5910b491-b4fd-45c1-97c6-549858854840 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:33.178920 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5910b491-b4fd-45c1-97c6-549858854840 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:33.179197 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5910b491-b4fd-45c1-97c6-549858854840 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:33.180917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 310/608] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:33.181847 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5910b491-b4fd-45c1-97c6-549858854840 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:33.188741 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd28d38-9065-4183-b4f7-c2112f9da0c5 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:33.225947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd85c9af-6bf3-4b4e-81d7-16647f9e3c1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:33.226573 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd85c9af-6bf3-4b4e-81d7-16647f9e3c1b None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:33.226901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd85c9af-6bf3-4b4e-81d7-16647f9e3c1b None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:33.227227 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd85c9af-6bf3-4b4e-81d7-16647f9e3c1b None admin] PATH_INFO: `/v3/projects/service` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:33.231591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:33.236392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd85c9af-6bf3-4b4e-81d7-16647f9e3c1b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:33.238525 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd85c9af-6bf3-4b4e-81d7-16647f9e3c1b None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:33.251470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-fd85c9af-6bf3-4b4e-81d7-16647f9e3c1b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:33.251470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:33.251470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:33.251470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:33.251470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:33.251470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:33.251470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:33.251470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:33.251470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:33.251470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:33.251470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:33.251470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:33.251470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:33.251470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:33.251470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:33.252617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:33.252617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:33.252617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:33.252617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:33.252617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:33.252617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:33.252617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:33.252617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:33.252617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:33.252617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:33.252617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:33.252617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:33.252617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:33.252617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:33.252617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:33.252617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:33.252617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:33.253821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:33.253821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:33.253821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:33.253821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:33.253821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:33.253821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:33.253821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:33.253821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:33.253821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 299/609] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:44:33.258537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9061234-a4fa-4d9f-8831-2a5298c75bfe None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:33.258815 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9061234-a4fa-4d9f-8831-2a5298c75bfe None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:33.259085 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9061234-a4fa-4d9f-8831-2a5298c75bfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:33.277360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-d9061234-a4fa-4d9f-8831-2a5298c75bfe None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:33.278542 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d9061234-a4fa-4d9f-8831-2a5298c75bfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:33.303386 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5910b491-b4fd-45c1-97c6-549858854840 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:33.319634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:33.344561 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b14ea92-6ec3-4947-898a-fd3d786d8006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:33.345167 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b14ea92-6ec3-4947-898a-fd3d786d8006 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:33.345444 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b14ea92-6ec3-4947-898a-fd3d786d8006 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:33.345712 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b14ea92-6ec3-4947-898a-fd3d786d8006 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:33.346342 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b14ea92-6ec3-4947-898a-fd3d786d8006 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:33.346639 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b14ea92-6ec3-4947-898a-fd3d786d8006 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:33.348155 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b14ea92-6ec3-4947-898a-fd3d786d8006 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:33.348341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 300/610] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:33 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:33.349199 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9061234-a4fa-4d9f-8831-2a5298c75bfe None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:33.368450 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 311/611] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:33.369339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b14ea92-6ec3-4947-898a-fd3d786d8006 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:33.466063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9061234-a4fa-4d9f-8831-2a5298c75bfe None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:33.469210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:33.472984 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b14ea92-6ec3-4947-898a-fd3d786d8006 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:33.510347 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f254a19e-831e-4aa0-a343-8b0bbf2f8141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:33.510989 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f254a19e-831e-4aa0-a343-8b0bbf2f8141 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:33.511323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f254a19e-831e-4aa0-a343-8b0bbf2f8141 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:33.511670 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f254a19e-831e-4aa0-a343-8b0bbf2f8141 None admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:33.512327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f254a19e-831e-4aa0-a343-8b0bbf2f8141 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:33.512634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f254a19e-831e-4aa0-a343-8b0bbf2f8141 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:33.513577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ef1f49-cfd0-41f9-a58b-4e1bc77cb08f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:33.514096 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f254a19e-831e-4aa0-a343-8b0bbf2f8141 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:33.514907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ef1f49-cfd0-41f9-a58b-4e1bc77cb08f None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:33.515674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ef1f49-cfd0-41f9-a58b-4e1bc77cb08f None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:33.516151 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ef1f49-cfd0-41f9-a58b-4e1bc77cb08f None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:33.517410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ef1f49-cfd0-41f9-a58b-4e1bc77cb08f None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:33.522762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ef1f49-cfd0-41f9-a58b-4e1bc77cb08f None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:33.527964 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 312/612] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:33.531806 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f254a19e-831e-4aa0-a343-8b0bbf2f8141 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:33.537962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-36ef1f49-cfd0-41f9-a58b-4e1bc77cb08f None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 31 14:44:33.537962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:33.537962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:33.537962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:33.537962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:33.537962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:33.537962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:33.537962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:33.537962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 31 14:44:33.537962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:33.537962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:33.537962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:33.537962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:33.537962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:33.537962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:33.537962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:33.537962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:33.538938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:33.538938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:33.538938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:33.538938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:33.538938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:33.538938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:33.538938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:33.538938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:33.538938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:33.538938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:33.538938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:33.538938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:33.538938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:33.538938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:33.538938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:33.538938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:33.539824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:33.539824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:33.539824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:33.539824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:33.539824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:33.539824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:33.539824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:33.539824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:33.539824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:33.539824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:33.539824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:33.539824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:33.539824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 31 14:44:33.539824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:33.543637 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 301/613] 10.222.0.36 () {56 vars in 1129 bytes} [Mon Oct 31 14:44:33 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:33.545214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ef1f49-cfd0-41f9-a58b-4e1bc77cb08f None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:33.604377 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:33.653967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f254a19e-831e-4aa0-a343-8b0bbf2f8141 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:33.689370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ef1f49-cfd0-41f9-a58b-4e1bc77cb08f None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:33.689756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7799677e-f951-4067-8aaf-f5f71653a89a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:33.690399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7799677e-f951-4067-8aaf-f5f71653a89a None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:33.690686 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7799677e-f951-4067-8aaf-f5f71653a89a None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:33.690972 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7799677e-f951-4067-8aaf-f5f71653a89a None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:33.699669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7799677e-f951-4067-8aaf-f5f71653a89a None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:33.701127 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7799677e-f951-4067-8aaf-f5f71653a89a None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:33.714403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-7799677e-f951-4067-8aaf-f5f71653a89a None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 31 14:44:33.714403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:33.714403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:33.714403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:33.714403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:33.714403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:33.714403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:33.714403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:33.714403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:33.714403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:33.714403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:33.714403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:33.714403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:33.714403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:33.714403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:33.714403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:33.715394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:33.716355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:33.716355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:33.716355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:33.716355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:33.716355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:33.716355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:33.716355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:33.716355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:33.716355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 31 14:44:33.716355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:33.721727 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 313/614] 10.222.0.36 () {56 vars in 1117 bytes} [Mon Oct 31 14:44:33 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:33.722769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7799677e-f951-4067-8aaf-f5f71653a89a None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:33.726958 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5800e3-b0e0-4ad1-9928-30bb0929f7ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:33.727580 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5800e3-b0e0-4ad1-9928-30bb0929f7ff None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:33.727878 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5800e3-b0e0-4ad1-9928-30bb0929f7ff None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:33.728152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5800e3-b0e0-4ad1-9928-30bb0929f7ff None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:33.728875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5800e3-b0e0-4ad1-9928-30bb0929f7ff None admin] RBAC: Adding query filter params (role.id=57572a37675d4c418e13794a7b8f3a5d, scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8, user.id=f4ae5e20cc5b40ccbdb951f908f5080d) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:33.729185 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5800e3-b0e0-4ad1-9928-30bb0929f7ff None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:33.731290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5800e3-b0e0-4ad1-9928-30bb0929f7ff None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:33.737718 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 302/615] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:44:33 2022] GET /identity/v3/role_assignments?user.id=f4ae5e20cc5b40ccbdb951f908f5080d&scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8&role.id=57572a37675d4c418e13794a7b8f3a5d => generated 608 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:33.738724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5800e3-b0e0-4ad1-9928-30bb0929f7ff None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:33.762804 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:33.868720 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7799677e-f951-4067-8aaf-f5f71653a89a None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:33.895710 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5800e3-b0e0-4ad1-9928-30bb0929f7ff None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:33.898223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:33.910561 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14c30fe-4dd4-4a78-a58d-3d21d4b75cc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:33.911435 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14c30fe-4dd4-4a78-a58d-3d21d4b75cc0 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:33.911904 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14c30fe-4dd4-4a78-a58d-3d21d4b75cc0 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:33.912340 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14c30fe-4dd4-4a78-a58d-3d21d4b75cc0 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:33.913118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e14c30fe-4dd4-4a78-a58d-3d21d4b75cc0 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:33.913567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e14c30fe-4dd4-4a78-a58d-3d21d4b75cc0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:33.915284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e14c30fe-4dd4-4a78-a58d-3d21d4b75cc0 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:33.923442 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 314/616] 10.222.0.36 () {56 vars in 1124 bytes} [Mon Oct 31 14:44:33 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:33.924666 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14c30fe-4dd4-4a78-a58d-3d21d4b75cc0 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:33.947331 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059285ba-f563-4485-821e-b8592c3a1f1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:33.948263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059285ba-f563-4485-821e-b8592c3a1f1d None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:33.948717 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059285ba-f563-4485-821e-b8592c3a1f1d None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:33.949084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059285ba-f563-4485-821e-b8592c3a1f1d None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:33.949651 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059285ba-f563-4485-821e-b8592c3a1f1d None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:33.950968 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059285ba-f563-4485-821e-b8592c3a1f1d None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:33.960727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-059285ba-f563-4485-821e-b8592c3a1f1d None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 31 14:44:33.960727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:33.960727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:33.960727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:33.960727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:33.960727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:33.960727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:33.960727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:33.960727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 31 14:44:33.960727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:33.960727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:33.960727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:33.960727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:33.960727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:33.960727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:33.960727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:33.961728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:33.961728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:33.961728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:33.961728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:33.961728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:33.961728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:33.961728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:33.961728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:33.961728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:33.961728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:33.961728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:33.961728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:33.961728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:33.961728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:33.961728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:33.961728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:33.961728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:33.963279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:33.963279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:33.963279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:33.963279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:33.963279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:33.963279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:33.963279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:33.963279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:33.963279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:33.963279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:33.963279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:33.963279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:33.963279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 31 14:44:33.963279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:33.963279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 303/617] 10.222.0.36 () {56 vars in 1132 bytes} [Mon Oct 31 14:44:33 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:33.964876 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059285ba-f563-4485-821e-b8592c3a1f1d None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:34.038613 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:34.062067 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14c30fe-4dd4-4a78-a58d-3d21d4b75cc0 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:34.083094 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059285ba-f563-4485-821e-b8592c3a1f1d None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:34.099045 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f184157-d336-4489-b9a6-d0a23be1b791 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:34.099695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f184157-d336-4489-b9a6-d0a23be1b791 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:34.099976 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f184157-d336-4489-b9a6-d0a23be1b791 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:34.100254 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f184157-d336-4489-b9a6-d0a23be1b791 None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:34.100983 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f184157-d336-4489-b9a6-d0a23be1b791 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:34.101287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f184157-d336-4489-b9a6-d0a23be1b791 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:34.102714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f184157-d336-4489-b9a6-d0a23be1b791 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:34.120951 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 315/618] 10.222.0.36 () {56 vars in 1149 bytes} [Mon Oct 31 14:44:33 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:34.122302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f184157-d336-4489-b9a6-d0a23be1b791 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:34.126331 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef79e91-3dcd-4a8d-b82e-c0281275d76c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:34.126977 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef79e91-3dcd-4a8d-b82e-c0281275d76c None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:34.127258 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef79e91-3dcd-4a8d-b82e-c0281275d76c None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:34.127529 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef79e91-3dcd-4a8d-b82e-c0281275d76c None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:34.128314 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef79e91-3dcd-4a8d-b82e-c0281275d76c None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:34.128736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef79e91-3dcd-4a8d-b82e-c0281275d76c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:34.130113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef79e91-3dcd-4a8d-b82e-c0281275d76c None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:34.140040 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 304/619] 10.222.0.36 () {56 vars in 1124 bytes} [Mon Oct 31 14:44:33 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:34.141360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96db92a4-c6af-436a-a4ca-a89cdb1408d4 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:34.141647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96db92a4-c6af-436a-a4ca-a89cdb1408d4 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:34.141921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96db92a4-c6af-436a-a4ca-a89cdb1408d4 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:34.147522 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 305/620] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:34 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:34.148328 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96db92a4-c6af-436a-a4ca-a89cdb1408d4 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:34.179947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:34.241782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f184157-d336-4489-b9a6-d0a23be1b791 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:34.256976 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96db92a4-c6af-436a-a4ca-a89cdb1408d4 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:34.278558 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d8fb0e-58a8-4791-9146-8cc5618d25e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:34.279170 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d8fb0e-58a8-4791-9146-8cc5618d25e8 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:34.279451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d8fb0e-58a8-4791-9146-8cc5618d25e8 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:34.279815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d8fb0e-58a8-4791-9146-8cc5618d25e8 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:34.280443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d8fb0e-58a8-4791-9146-8cc5618d25e8 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:34.280831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d8fb0e-58a8-4791-9146-8cc5618d25e8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:34.282272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d8fb0e-58a8-4791-9146-8cc5618d25e8 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:34.293635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 316/621] 10.222.0.36 () {56 vars in 1126 bytes} [Mon Oct 31 14:44:34 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:34.293635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d8fb0e-58a8-4791-9146-8cc5618d25e8 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:34.297252 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3ebc1e-df3e-4dd7-8e03-463f8610c6ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:34.297865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3ebc1e-df3e-4dd7-8e03-463f8610c6ae None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:34.298162 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3ebc1e-df3e-4dd7-8e03-463f8610c6ae None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:34.298418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3ebc1e-df3e-4dd7-8e03-463f8610c6ae None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:34.298612 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:34.374754 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3ebc1e-df3e-4dd7-8e03-463f8610c6ae None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:34.376946 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3ebc1e-df3e-4dd7-8e03-463f8610c6ae None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:34.397391 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d8fb0e-58a8-4791-9146-8cc5618d25e8 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:34.411578 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:34.434515 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98339764-ff08-473a-bf3e-846587e90043 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:34.435120 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98339764-ff08-473a-bf3e-846587e90043 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:34.435395 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98339764-ff08-473a-bf3e-846587e90043 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:34.435695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98339764-ff08-473a-bf3e-846587e90043 None admin] PATH_INFO: `/v3/endpoints` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:34.436358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98339764-ff08-473a-bf3e-846587e90043 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:34.437703 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98339764-ff08-473a-bf3e-846587e90043 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:34.461728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 317/622] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:44:34 2022] POST /identity/v3/endpoints => generated 346 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:34.462800 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8f8110-050c-444d-a580-159c6920a3b8 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:34.463090 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8f8110-050c-444d-a580-159c6920a3b8 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:34.463420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8f8110-050c-444d-a580-159c6920a3b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:34.488072 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-2c8f8110-050c-444d-a580-159c6920a3b8 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:34.489761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2c8f8110-050c-444d-a580-159c6920a3b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:34.515119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 306/623] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:44:34 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 31 14:44:34.515119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3ebc1e-df3e-4dd7-8e03-463f8610c6ae None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:34.573171 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:34.595992 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 318/624] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:34 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:34.597494 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8f8110-050c-444d-a580-159c6920a3b8 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:34.650411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3ebc1e-df3e-4dd7-8e03-463f8610c6ae None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:34.685399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:34.693104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a33bac-4831-4bae-a761-078f96addec1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:34.693721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a33bac-4831-4bae-a761-078f96addec1 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:34.694010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a33bac-4831-4bae-a761-078f96addec1 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:34.694286 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a33bac-4831-4bae-a761-078f96addec1 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:34.694921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a33bac-4831-4bae-a761-078f96addec1 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:34.695355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a33bac-4831-4bae-a761-078f96addec1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:34.696901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a33bac-4831-4bae-a761-078f96addec1 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:34.702665 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 307/625] 10.222.0.36 () {56 vars in 1126 bytes} [Mon Oct 31 14:44:34 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:34.716186 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8f8110-050c-444d-a580-159c6920a3b8 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:34.759410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f91c328-3215-4b9a-a94f-3cfd1acb9623 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:34.760090 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f91c328-3215-4b9a-a94f-3cfd1acb9623 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:34.760369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f91c328-3215-4b9a-a94f-3cfd1acb9623 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:34.760642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f91c328-3215-4b9a-a94f-3cfd1acb9623 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:34.773550 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f91c328-3215-4b9a-a94f-3cfd1acb9623 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:34.775045 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f91c328-3215-4b9a-a94f-3cfd1acb9623 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:34.791986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-7f91c328-3215-4b9a-a94f-3cfd1acb9623 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:34.791986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:34.791986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:34.791986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:34.791986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:34.791986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:34.791986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:34.791986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:34.791986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:34.791986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:34.791986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:34.791986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:34.791986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:34.791986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:34.791986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:34.791986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:34.793235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:34.793235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:34.793235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:34.793235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:34.793235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:34.793235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:34.793235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:34.793235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:34.793235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:34.793235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:34.793235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:34.793235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:34.793235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:34.793235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:34.793235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:34.793235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:34.793235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:34.794127 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:34.794127 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:34.794127 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:34.794127 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 319/626] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:34.800536 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a33bac-4831-4bae-a761-078f96addec1 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:34.916036 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a33bac-4831-4bae-a761-078f96addec1 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:34.921871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f91c328-3215-4b9a-a94f-3cfd1acb9623 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:34.962413 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25504d0-5a4b-4a90-8eb0-d9d7360b1d80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:34.963024 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25504d0-5a4b-4a90-8eb0-d9d7360b1d80 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:34.963299 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25504d0-5a4b-4a90-8eb0-d9d7360b1d80 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:34.963584 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25504d0-5a4b-4a90-8eb0-d9d7360b1d80 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:34.964270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25504d0-5a4b-4a90-8eb0-d9d7360b1d80 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:34.964575 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25504d0-5a4b-4a90-8eb0-d9d7360b1d80 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:34.965946 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25504d0-5a4b-4a90-8eb0-d9d7360b1d80 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:34.978798 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:34.979195 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 308/627] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:34.985949 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25504d0-5a4b-4a90-8eb0-d9d7360b1d80 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:35.066741 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f91c328-3215-4b9a-a94f-3cfd1acb9623 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:35.103295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:35.124541 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e286d044-a2e7-4e4d-ac79-a33878f5d34a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:35.125181 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e286d044-a2e7-4e4d-ac79-a33878f5d34a None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:35.125461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e286d044-a2e7-4e4d-ac79-a33878f5d34a None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:35.125738 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e286d044-a2e7-4e4d-ac79-a33878f5d34a None admin] PATH_INFO: `/v3/services/network` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:35.126327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e286d044-a2e7-4e4d-ac79-a33878f5d34a None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:35.127797 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e286d044-a2e7-4e4d-ac79-a33878f5d34a None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:35.143903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-e286d044-a2e7-4e4d-ac79-a33878f5d34a None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 31 14:44:35.143903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:35.143903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:35.143903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:35.143903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:35.143903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:35.143903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:35.143903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:35.143903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 31 14:44:35.143903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:35.143903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:35.143903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:35.143903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:35.143903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:35.143903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:35.143903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:35.143903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:35.144954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:35.144954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:35.144954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:35.144954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:35.144954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:35.144954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:35.144954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:35.144954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:35.144954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:35.144954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:35.144954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:35.144954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:35.144954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:35.144954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:35.144954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:35.144954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:35.145783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:35.145783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:35.145783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:35.145783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:35.145783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:35.145783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:35.145783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:35.145783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:35.145783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:35.145783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:35.145783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:35.145783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:35.145783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 31 14:44:35.145783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:35.145783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 320/628] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:34 2022] GET /identity/v3/services/network => generated 88 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:35.155308 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e286d044-a2e7-4e4d-ac79-a33878f5d34a None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:35.193899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25504d0-5a4b-4a90-8eb0-d9d7360b1d80 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:35.264521 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969e7b37-a42c-4e83-a55b-2e65388cc03e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:35.265159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969e7b37-a42c-4e83-a55b-2e65388cc03e None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:35.265621 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969e7b37-a42c-4e83-a55b-2e65388cc03e None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:35.265903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969e7b37-a42c-4e83-a55b-2e65388cc03e None admin] PATH_INFO: `/v3/projects/service` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:35.273654 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e286d044-a2e7-4e4d-ac79-a33878f5d34a None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:35.275467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:35.285054 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969e7b37-a42c-4e83-a55b-2e65388cc03e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:35.286521 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969e7b37-a42c-4e83-a55b-2e65388cc03e None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:35.303348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-969e7b37-a42c-4e83-a55b-2e65388cc03e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:35.303348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:35.303348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:35.303348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:35.303348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:35.303348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:35.303348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:35.303348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:35.303348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:35.303348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:35.303348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:35.303348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:35.303348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:35.303348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:35.303348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:35.304289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:35.304289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:35.304289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:35.304289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:35.304289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:35.304289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:35.304289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:35.304289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:35.304289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:35.304289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:35.304289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:35.304289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:35.304289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:35.304289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:35.304289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:35.304289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:35.304289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:35.305244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:35.305244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:35.305244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:35.305244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:35.305244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:35.305244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:35.305244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:35.305244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:35.305244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 309/629] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:35.310845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969e7b37-a42c-4e83-a55b-2e65388cc03e None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:35.315819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d71f92-9d17-4b86-bf7e-a769ee9ac7d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:35.316663 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d71f92-9d17-4b86-bf7e-a769ee9ac7d4 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:35.317170 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d71f92-9d17-4b86-bf7e-a769ee9ac7d4 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:35.317633 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d71f92-9d17-4b86-bf7e-a769ee9ac7d4 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:35.318756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d71f92-9d17-4b86-bf7e-a769ee9ac7d4 None admin] RBAC: Adding query filter params (name=network) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:35.319256 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d71f92-9d17-4b86-bf7e-a769ee9ac7d4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:35.321775 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d71f92-9d17-4b86-bf7e-a769ee9ac7d4 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:35.329908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 321/630] 10.222.0.36 () {56 vars in 1112 bytes} [Mon Oct 31 14:44:35 2022] GET /identity/v3/services?name=network => generated 124 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:35.335946 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d71f92-9d17-4b86-bf7e-a769ee9ac7d4 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:35.473559 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d71f92-9d17-4b86-bf7e-a769ee9ac7d4 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:35.537905 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969e7b37-a42c-4e83-a55b-2e65388cc03e None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:35.552279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0225ca-ba21-4b37-ad16-7df06ca86704 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:35.552889 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0225ca-ba21-4b37-ad16-7df06ca86704 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:35.553164 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0225ca-ba21-4b37-ad16-7df06ca86704 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:35.553434 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0225ca-ba21-4b37-ad16-7df06ca86704 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:35.554057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0225ca-ba21-4b37-ad16-7df06ca86704 None admin] RBAC: Adding query filter params (type=network) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:35.554357 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0225ca-ba21-4b37-ad16-7df06ca86704 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:35.555824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0225ca-ba21-4b37-ad16-7df06ca86704 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:35.566635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 322/631] 10.222.0.36 () {56 vars in 1112 bytes} [Mon Oct 31 14:44:35 2022] GET /identity/v3/services?type=network => generated 350 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:35.574507 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:35.575648 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0225ca-ba21-4b37-ad16-7df06ca86704 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:35.617919 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6a313e-4c66-4c2c-b05b-62290df19e61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:35.618526 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6a313e-4c66-4c2c-b05b-62290df19e61 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:35.618803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6a313e-4c66-4c2c-b05b-62290df19e61 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:35.619071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6a313e-4c66-4c2c-b05b-62290df19e61 None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:35.620838 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6a313e-4c66-4c2c-b05b-62290df19e61 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:35.620838 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6a313e-4c66-4c2c-b05b-62290df19e61 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:35.622059 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6a313e-4c66-4c2c-b05b-62290df19e61 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:35.641237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:35.649791 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 310/632] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 340 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:35.656005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6a313e-4c66-4c2c-b05b-62290df19e61 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:35.727324 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0225ca-ba21-4b37-ad16-7df06ca86704 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:35.765856 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b91755e-b4b0-4b21-87b1-d645e77e515f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:35.766827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b91755e-b4b0-4b21-87b1-d645e77e515f None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:35.767313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b91755e-b4b0-4b21-87b1-d645e77e515f None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:35.767823 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b91755e-b4b0-4b21-87b1-d645e77e515f None admin] PATH_INFO: `/v3/endpoints` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:35.768416 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6a313e-4c66-4c2c-b05b-62290df19e61 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:35.768986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b91755e-b4b0-4b21-87b1-d645e77e515f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:35.770809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b91755e-b4b0-4b21-87b1-d645e77e515f None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:35.794266 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 323/633] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:44:35 2022] POST /identity/v3/endpoints => generated 337 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:35.827773 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907489d8-f02f-44d0-9da9-110d07dd9224 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:35.828451 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907489d8-f02f-44d0-9da9-110d07dd9224 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:35.828745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907489d8-f02f-44d0-9da9-110d07dd9224 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:35.829022 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907489d8-f02f-44d0-9da9-110d07dd9224 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:35.829848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907489d8-f02f-44d0-9da9-110d07dd9224 None admin] RBAC: Adding query filter params (role.id=88d62c66100a4d04a45e3607135bdd0c, scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8, user.id=e0d0f8598fac40449bcb2a55ff8df8d9) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:35.830209 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907489d8-f02f-44d0-9da9-110d07dd9224 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:35.831828 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907489d8-f02f-44d0-9da9-110d07dd9224 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:35.845536 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 311/634] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:44:35 2022] GET /identity/v3/role_assignments?user.id=e0d0f8598fac40449bcb2a55ff8df8d9&scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8&role.id=88d62c66100a4d04a45e3607135bdd0c => generated 608 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:36.116740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ebdb4b-c4bd-4300-a473-f576be2726c7 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:36.117719 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ebdb4b-c4bd-4300-a473-f576be2726c7 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:36.118258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ebdb4b-c4bd-4300-a473-f576be2726c7 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:36.120101 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 324/635] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:36.126519 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8d86e6-ef57-4f26-bbce-8d7df2f58619 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:36.126795 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8d86e6-ef57-4f26-bbce-8d7df2f58619 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:36.127071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8d86e6-ef57-4f26-bbce-8d7df2f58619 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:36.151467 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-6a8d86e6-ef57-4f26-bbce-8d7df2f58619 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:36.152877 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6a8d86e6-ef57-4f26-bbce-8d7df2f58619 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:36.188484 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:36.235642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 312/636] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:36 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 31 14:44:36.515155 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ebdb4b-c4bd-4300-a473-f576be2726c7 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:36.618775 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:36.635779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ebdb4b-c4bd-4300-a473-f576be2726c7 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:36.671908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d28aca-3e99-4d89-a2a3-3b1e03fde7b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:36.672938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d28aca-3e99-4d89-a2a3-3b1e03fde7b8 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:36.673423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d28aca-3e99-4d89-a2a3-3b1e03fde7b8 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:36.673917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d28aca-3e99-4d89-a2a3-3b1e03fde7b8 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:36.674700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d28aca-3e99-4d89-a2a3-3b1e03fde7b8 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:36.676384 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d28aca-3e99-4d89-a2a3-3b1e03fde7b8 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:36.685843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-65d28aca-3e99-4d89-a2a3-3b1e03fde7b8 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 31 14:44:36.685843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:36.685843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:36.685843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:36.685843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:36.685843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:36.685843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:36.685843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:36.685843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 31 14:44:36.685843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:36.685843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:36.685843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:36.685843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:36.685843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:36.685843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:36.685843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:36.685843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:36.686830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:36.686830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:36.686830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:36.686830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:36.686830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:36.686830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:36.686830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:36.686830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:36.686830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:36.686830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:36.686830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:36.686830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:36.686830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:36.686830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:36.686830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:36.686830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:36.687693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:36.687693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:36.687693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:36.687693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:36.687693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:36.687693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:36.687693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:36.687693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:36.687693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:36.687693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:36.687693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:36.687693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:36.687693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 31 14:44:36.687693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:36.690410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 325/637] 10.222.0.36 () {56 vars in 1126 bytes} [Mon Oct 31 14:44:36 2022] GET /identity/v3/services/object-store => generated 93 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:36.696529 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8d86e6-ef57-4f26-bbce-8d7df2f58619 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:36.804892 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8d86e6-ef57-4f26-bbce-8d7df2f58619 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:36.842132 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66ddd19-fe5c-4ffc-bc14-0ec2d9ca4109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:36.843141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66ddd19-fe5c-4ffc-bc14-0ec2d9ca4109 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:36.843923 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66ddd19-fe5c-4ffc-bc14-0ec2d9ca4109 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:36.844398 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66ddd19-fe5c-4ffc-bc14-0ec2d9ca4109 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:36.845238 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66ddd19-fe5c-4ffc-bc14-0ec2d9ca4109 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:36.845746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66ddd19-fe5c-4ffc-bc14-0ec2d9ca4109 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:36.847369 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66ddd19-fe5c-4ffc-bc14-0ec2d9ca4109 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:36.852653 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 313/638] 10.222.0.36 () {56 vars in 1122 bytes} [Mon Oct 31 14:44:36 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:36.858767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d28aca-3e99-4d89-a2a3-3b1e03fde7b8 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:36.965032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d28aca-3e99-4d89-a2a3-3b1e03fde7b8 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:37.003777 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d87b938-5133-4a12-9fa3-5c8dfcd7ccc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:37.004782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d87b938-5133-4a12-9fa3-5c8dfcd7ccc4 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:37.005283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d87b938-5133-4a12-9fa3-5c8dfcd7ccc4 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:37.005760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d87b938-5133-4a12-9fa3-5c8dfcd7ccc4 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:37.006589 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d87b938-5133-4a12-9fa3-5c8dfcd7ccc4 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:37.007093 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d87b938-5133-4a12-9fa3-5c8dfcd7ccc4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:37.008750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d87b938-5133-4a12-9fa3-5c8dfcd7ccc4 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:37.014952 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 326/639] 10.222.0.36 () {56 vars in 1122 bytes} [Mon Oct 31 14:44:36 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:37.152980 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb5c173-8bd5-42a3-95c4-a133c6047a04 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:37.153171 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb5c173-8bd5-42a3-95c4-a133c6047a04 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:37.153427 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb5c173-8bd5-42a3-95c4-a133c6047a04 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:37.155069 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 314/640] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:37.160960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492cb134-ca2c-4112-9313-3247bf6070a5 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:37.161497 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492cb134-ca2c-4112-9313-3247bf6070a5 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:37.161575 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492cb134-ca2c-4112-9313-3247bf6070a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:37.181769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-492cb134-ca2c-4112-9313-3247bf6070a5 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:37.183585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-492cb134-ca2c-4112-9313-3247bf6070a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:37.194275 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:37.261919 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 327/641] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:37 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:37.340203 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e25cfb2-f74d-4c44-91f1-4ebd16f8e479 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:37.340535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e25cfb2-f74d-4c44-91f1-4ebd16f8e479 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:37.340753 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e25cfb2-f74d-4c44-91f1-4ebd16f8e479 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:37.342357 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 315/642] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:37.347905 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5faeb9e-1fbb-476a-b5f6-23dd5a0235ab None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:37.348193 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5faeb9e-1fbb-476a-b5f6-23dd5a0235ab None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:37.348466 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5faeb9e-1fbb-476a-b5f6-23dd5a0235ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:37.367529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-e5faeb9e-1fbb-476a-b5f6-23dd5a0235ab None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:37.368760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e5faeb9e-1fbb-476a-b5f6-23dd5a0235ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:37.436694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 328/643] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:37 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:37.544920 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e25cfb2-f74d-4c44-91f1-4ebd16f8e479 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:37.574513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:37.653636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e25cfb2-f74d-4c44-91f1-4ebd16f8e479 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:37.690816 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87353176-f8fc-4fe4-9de0-7494bbc450a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:37.691790 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87353176-f8fc-4fe4-9de0-7494bbc450a8 None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:37.692279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87353176-f8fc-4fe4-9de0-7494bbc450a8 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:37.692753 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87353176-f8fc-4fe4-9de0-7494bbc450a8 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:37.693608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87353176-f8fc-4fe4-9de0-7494bbc450a8 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:37.695166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87353176-f8fc-4fe4-9de0-7494bbc450a8 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:37.707250 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5faeb9e-1fbb-476a-b5f6-23dd5a0235ab None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:37.714944 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 316/644] 10.222.0.36 () {60 vars in 1141 bytes} [Mon Oct 31 14:44:37 2022] POST /identity/v3/services => generated 269 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:37.803767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:37.833021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5faeb9e-1fbb-476a-b5f6-23dd5a0235ab None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:37.880519 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f876f617-adac-42e0-9135-132c7a8b00f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:37.881595 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f876f617-adac-42e0-9135-132c7a8b00f1 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:37.882082 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f876f617-adac-42e0-9135-132c7a8b00f1 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:37.882567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f876f617-adac-42e0-9135-132c7a8b00f1 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:37.883361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f876f617-adac-42e0-9135-132c7a8b00f1 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:37.885002 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f876f617-adac-42e0-9135-132c7a8b00f1 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:37.895424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-f876f617-adac-42e0-9135-132c7a8b00f1 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 31 14:44:37.895424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:37.895424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:37.895424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:37.895424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:37.895424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:37.895424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:37.895424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:37.895424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 31 14:44:37.895424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:37.895424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:37.895424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:37.895424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:37.895424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:37.895424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:37.895424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:37.895424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:37.896521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:37.896521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:37.896521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:37.896521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:37.896521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:37.896521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:37.896521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:37.896521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:37.896521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:37.896521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:37.896521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:37.896521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:37.896521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:37.896521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:37.896521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:37.896521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:37.897390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:37.897390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:37.897390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:37.897390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:37.897390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:37.897390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:37.897390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:37.897390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:37.897390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:37.897390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:37.897390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:37.897390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:37.897390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 31 14:44:37.897390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:37.900295 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 329/645] 10.222.0.36 () {56 vars in 1114 bytes} [Mon Oct 31 14:44:37 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:37.907524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87353176-f8fc-4fe4-9de0-7494bbc450a8 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:38.014057 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:38.028264 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87353176-f8fc-4fe4-9de0-7494bbc450a8 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:38.068012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40360269-1615-4fb4-9e7c-95ed9741436c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:38.068985 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40360269-1615-4fb4-9e7c-95ed9741436c None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:38.069471 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40360269-1615-4fb4-9e7c-95ed9741436c None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:38.069991 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40360269-1615-4fb4-9e7c-95ed9741436c None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:38.070828 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40360269-1615-4fb4-9e7c-95ed9741436c None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:38.071332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40360269-1615-4fb4-9e7c-95ed9741436c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:38.072942 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40360269-1615-4fb4-9e7c-95ed9741436c None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:38.079047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 317/646] 10.222.0.36 () {56 vars in 1114 bytes} [Mon Oct 31 14:44:37 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:38.084742 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f876f617-adac-42e0-9135-132c7a8b00f1 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:38.193154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f876f617-adac-42e0-9135-132c7a8b00f1 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:38.227544 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72d14d8-8182-425b-b123-a88001db730e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:38.228583 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72d14d8-8182-425b-b123-a88001db730e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:38.229086 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72d14d8-8182-425b-b123-a88001db730e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:38.230161 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72d14d8-8182-425b-b123-a88001db730e None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:38.231085 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72d14d8-8182-425b-b123-a88001db730e None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:38.231616 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72d14d8-8182-425b-b123-a88001db730e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:38.233219 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72d14d8-8182-425b-b123-a88001db730e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:38.235145 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:38.239629 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 330/647] 10.222.0.36 () {56 vars in 1114 bytes} [Mon Oct 31 14:44:38 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:38.363730 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0357cd79-b16c-4942-8085-0bea4e0c5644 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:38.363916 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0357cd79-b16c-4942-8085-0bea4e0c5644 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:38.364185 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0357cd79-b16c-4942-8085-0bea4e0c5644 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:38.365884 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 318/648] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:38.372907 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2294e4b2-adc2-4972-82c5-0e98bbd3a5da None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:38.373247 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2294e4b2-adc2-4972-82c5-0e98bbd3a5da None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:38.373460 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2294e4b2-adc2-4972-82c5-0e98bbd3a5da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:38.393594 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-2294e4b2-adc2-4972-82c5-0e98bbd3a5da None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:38.394843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2294e4b2-adc2-4972-82c5-0e98bbd3a5da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:38.469520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 331/649] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:38 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:38.745749 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0357cd79-b16c-4942-8085-0bea4e0c5644 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:38.859324 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0357cd79-b16c-4942-8085-0bea4e0c5644 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:38.898204 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cbe261-c0e5-4bb5-be4b-cab2f23e5142 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:38.898896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cbe261-c0e5-4bb5-be4b-cab2f23e5142 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:38.899174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cbe261-c0e5-4bb5-be4b-cab2f23e5142 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:38.899450 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cbe261-c0e5-4bb5-be4b-cab2f23e5142 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:38.911309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cbe261-c0e5-4bb5-be4b-cab2f23e5142 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:38.916335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-40cbe261-c0e5-4bb5-be4b-cab2f23e5142 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 31 14:44:38.916335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:38.916335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:38.916335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:38.916335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:38.916335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:38.916335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:38.916335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:38.916335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:38.916335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:38.916335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:38.916335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:38.916335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:38.916335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:38.916335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:38.916335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:38.917379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:38.917379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:38.917379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:38.917379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:38.917379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:38.917379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:38.917379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:38.917379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:38.917379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:38.917379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:38.917379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:38.917379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:38.917379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:38.917379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:38.917379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:38.917379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:38.918213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:38.918213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:38.918213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:38.918213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:38.918213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:38.918213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:38.918213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:38.918213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:38.918213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:38.918213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:38.918213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 31 14:44:38.918213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:38.918213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 319/650] 10.222.0.36 () {56 vars in 1120 bytes} [Mon Oct 31 14:44:38 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:38.924917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2294e4b2-adc2-4972-82c5-0e98bbd3a5da None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:39.033301 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2294e4b2-adc2-4972-82c5-0e98bbd3a5da None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:39.060952 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:39.070879 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b32253-7cca-47e0-8f47-5253051d9577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:39.071827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b32253-7cca-47e0-8f47-5253051d9577 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:39.072304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b32253-7cca-47e0-8f47-5253051d9577 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:39.072767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b32253-7cca-47e0-8f47-5253051d9577 None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:39.074170 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b32253-7cca-47e0-8f47-5253051d9577 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:39.074685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b32253-7cca-47e0-8f47-5253051d9577 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:39.076376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b32253-7cca-47e0-8f47-5253051d9577 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:39.085349 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 332/651] 10.222.0.36 () {56 vars in 1115 bytes} [Mon Oct 31 14:44:38 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:39.091985 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cbe261-c0e5-4bb5-be4b-cab2f23e5142 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:39.108519 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:39.213101 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cbe261-c0e5-4bb5-be4b-cab2f23e5142 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:39.251428 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cbdf5d-12f4-4408-9b62-323f0b7bbe11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:39.252085 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cbdf5d-12f4-4408-9b62-323f0b7bbe11 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:39.252362 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cbdf5d-12f4-4408-9b62-323f0b7bbe11 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:39.252635 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cbdf5d-12f4-4408-9b62-323f0b7bbe11 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:39.263824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5cbdf5d-12f4-4408-9b62-323f0b7bbe11 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:39.265364 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5cbdf5d-12f4-4408-9b62-323f0b7bbe11 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:39.277861 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb963d9-02b7-45e4-8f63-10c382cb1266 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:39.278373 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb963d9-02b7-45e4-8f63-10c382cb1266 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:39.278373 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb963d9-02b7-45e4-8f63-10c382cb1266 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:39.280156 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 333/652] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:39.280992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-f5cbdf5d-12f4-4408-9b62-323f0b7bbe11 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:39.280992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:39.280992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:39.280992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:39.280992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:39.280992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:39.280992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:39.280992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:39.280992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:39.280992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:39.280992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:39.280992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:39.280992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:39.280992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:39.280992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:39.280992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:39.282044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:39.282044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:39.282044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:39.282044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:39.282044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:39.282044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:39.282044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:39.282044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:39.282044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:39.282044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:39.282044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:39.282044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:39.282044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:39.282044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:39.282044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:39.282044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:39.282044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:39.283239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:39.283239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:39.283239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:39.283239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 320/653] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:39.285133 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6736e9d-646a-4078-97c6-d53eb7049634 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:39.285362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6736e9d-646a-4078-97c6-d53eb7049634 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:39.285635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6736e9d-646a-4078-97c6-d53eb7049634 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:39.290114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cbdf5d-12f4-4408-9b62-323f0b7bbe11 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:39.304526 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-b6736e9d-646a-4078-97c6-d53eb7049634 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:39.305711 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b6736e9d-646a-4078-97c6-d53eb7049634 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:39.360129 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:39.386340 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 334/654] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:39 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:39.411821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cbdf5d-12f4-4408-9b62-323f0b7bbe11 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:39.446300 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2be7610-e5c0-4357-b235-0ad2513df7f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:39.446890 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2be7610-e5c0-4357-b235-0ad2513df7f8 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:39.447165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2be7610-e5c0-4357-b235-0ad2513df7f8 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:39.447437 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2be7610-e5c0-4357-b235-0ad2513df7f8 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:39.448680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2be7610-e5c0-4357-b235-0ad2513df7f8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:39.449331 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2be7610-e5c0-4357-b235-0ad2513df7f8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:39.450620 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2be7610-e5c0-4357-b235-0ad2513df7f8 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:39.463751 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 321/655] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:39.471038 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6736e9d-646a-4078-97c6-d53eb7049634 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:39.574538 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6736e9d-646a-4078-97c6-d53eb7049634 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:39.595901 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:39.611998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a902e8-c0ed-4606-9766-2a6085b97913 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:39.612628 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a902e8-c0ed-4606-9766-2a6085b97913 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:39.612905 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a902e8-c0ed-4606-9766-2a6085b97913 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:39.613175 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a902e8-c0ed-4606-9766-2a6085b97913 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:39.624649 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a902e8-c0ed-4606-9766-2a6085b97913 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:39.626361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a902e8-c0ed-4606-9766-2a6085b97913 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:39.639051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-92a902e8-c0ed-4606-9766-2a6085b97913 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 31 14:44:39.639051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:39.639051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:39.639051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:39.639051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:39.639051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:39.639051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:39.639051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:39.639051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:39.639051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:39.639051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:39.639051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:39.639051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:39.639051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:39.639051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:39.639051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:39.640104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:39.641039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:39.641039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:39.641039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:39.641039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:39.641039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:39.641039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:39.641039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:39.641039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:39.641039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 31 14:44:39.641039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:39.647652 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 335/656] 10.222.0.36 () {56 vars in 1117 bytes} [Mon Oct 31 14:44:39 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:44:39.654228 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2be7610-e5c0-4357-b235-0ad2513df7f8 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:39.657181 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a902e8-c0ed-4606-9766-2a6085b97913 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:39.774000 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a902e8-c0ed-4606-9766-2a6085b97913 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:39.782403 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2be7610-e5c0-4357-b235-0ad2513df7f8 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:39.811822 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:39.826674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca1d709-c2cd-4902-9251-4e158d7ee3f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:39.827810 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca1d709-c2cd-4902-9251-4e158d7ee3f4 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:39.828418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca1d709-c2cd-4902-9251-4e158d7ee3f4 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:39.828944 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca1d709-c2cd-4902-9251-4e158d7ee3f4 None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:39.829843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca1d709-c2cd-4902-9251-4e158d7ee3f4 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:39.830294 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca1d709-c2cd-4902-9251-4e158d7ee3f4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:39.832388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca1d709-c2cd-4902-9251-4e158d7ee3f4 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:39.842210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d98c72-a8cd-46a5-9851-597da89d14c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:39.843839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d98c72-a8cd-46a5-9851-597da89d14c4 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:39.844522 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d98c72-a8cd-46a5-9851-597da89d14c4 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:39.845197 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d98c72-a8cd-46a5-9851-597da89d14c4 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:39.846566 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d98c72-a8cd-46a5-9851-597da89d14c4 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:39.849317 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d98c72-a8cd-46a5-9851-597da89d14c4 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:39.852459 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 322/657] 10.222.0.36 () {56 vars in 1149 bytes} [Mon Oct 31 14:44:39 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:39.859203 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca1d709-c2cd-4902-9251-4e158d7ee3f4 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:39.878419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 336/658] 10.222.0.36 () {60 vars in 1141 bytes} [Mon Oct 31 14:44:39 2022] POST /identity/v3/services => generated 241 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:44:39.991773 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca1d709-c2cd-4902-9251-4e158d7ee3f4 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:40.040863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a246af-309f-4e52-90f2-da650ee953f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:40.041889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a246af-309f-4e52-90f2-da650ee953f1 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:40.042919 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a246af-309f-4e52-90f2-da650ee953f1 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:40.043384 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a246af-309f-4e52-90f2-da650ee953f1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:40.055904 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a246af-309f-4e52-90f2-da650ee953f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:40.058843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a246af-309f-4e52-90f2-da650ee953f1 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:40.076289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-77a246af-309f-4e52-90f2-da650ee953f1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:40.076289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:40.076289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:40.076289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:40.076289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:40.076289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:40.076289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:40.076289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:40.076289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:40.076289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:40.076289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:40.076289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:40.076289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:40.076289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:40.076289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:40.076289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:40.077306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:40.077306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:40.077306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:40.077306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:40.077306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:40.077306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:40.077306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:40.077306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:40.077306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:40.077306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:40.077306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:40.077306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:40.077306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:40.077306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:40.077306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:40.077306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:40.077306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:40.078259 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:40.078259 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:40.078259 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:40.078982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 323/659] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:40.084686 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d98c72-a8cd-46a5-9851-597da89d14c4 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:40.136237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:40.203055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d98c72-a8cd-46a5-9851-597da89d14c4 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:40.239443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11931d1f-b45c-45d4-9ce1-b5b803e79155 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:40.240524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11931d1f-b45c-45d4-9ce1-b5b803e79155 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:40.241006 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11931d1f-b45c-45d4-9ce1-b5b803e79155 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:40.241477 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11931d1f-b45c-45d4-9ce1-b5b803e79155 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:40.242343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11931d1f-b45c-45d4-9ce1-b5b803e79155 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:40.242841 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11931d1f-b45c-45d4-9ce1-b5b803e79155 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:40.244519 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11931d1f-b45c-45d4-9ce1-b5b803e79155 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:40.249389 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637e0904-a56c-4934-88d1-a1881ba7ec64 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:40.249674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637e0904-a56c-4934-88d1-a1881ba7ec64 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:40.249949 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637e0904-a56c-4934-88d1-a1881ba7ec64 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:40.251537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 324/660] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:40.256689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5485234-df26-40e9-928e-0a99341356ba None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:40.256899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5485234-df26-40e9-928e-0a99341356ba None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:40.257165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5485234-df26-40e9-928e-0a99341356ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:40.259639 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 337/661] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:40.265180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11931d1f-b45c-45d4-9ce1-b5b803e79155 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:40.276847 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-d5485234-df26-40e9-928e-0a99341356ba None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:40.278386 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d5485234-df26-40e9-928e-0a99341356ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:40.304021 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:40.353443 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 325/662] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:40 2022] POST /identity/v3/auth/tokens => generated 1373 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 31 14:44:40.376521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:40.380635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11931d1f-b45c-45d4-9ce1-b5b803e79155 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:40.421432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe58d4bb-5052-46b3-9e21-d81c59fd7dc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:40.422126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe58d4bb-5052-46b3-9e21-d81c59fd7dc1 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:40.422405 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe58d4bb-5052-46b3-9e21-d81c59fd7dc1 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:40.422677 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe58d4bb-5052-46b3-9e21-d81c59fd7dc1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:40.429769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe58d4bb-5052-46b3-9e21-d81c59fd7dc1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:40.431207 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe58d4bb-5052-46b3-9e21-d81c59fd7dc1 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:40.444514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-fe58d4bb-5052-46b3-9e21-d81c59fd7dc1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:40.444514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:40.444514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:40.444514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:40.444514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:40.444514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:40.444514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:40.444514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:40.444514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:40.444514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:40.444514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:40.444514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:40.444514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:40.444514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:40.444514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:40.445432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:40.445432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:40.445432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:40.445432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:40.445432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:40.445432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:40.445432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:40.445432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:40.445432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:40.445432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:40.445432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:40.445432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:40.445432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:40.445432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:40.445432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:40.445432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:40.445432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:40.446696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:40.446696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:40.446696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:40.446696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:40.446696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:40.446696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:40.446696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:40.446696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:40.446696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 338/663] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:40.453293 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5485234-df26-40e9-928e-0a99341356ba None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:40.572186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5485234-df26-40e9-928e-0a99341356ba None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:40.611139 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2ea592-6055-4cb4-ba2a-54b02f8e90dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:40.611721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2ea592-6055-4cb4-ba2a-54b02f8e90dd None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:40.612002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2ea592-6055-4cb4-ba2a-54b02f8e90dd None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:40.612276 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2ea592-6055-4cb4-ba2a-54b02f8e90dd None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:40.612999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2ea592-6055-4cb4-ba2a-54b02f8e90dd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:40.613383 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2ea592-6055-4cb4-ba2a-54b02f8e90dd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:40.614882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2ea592-6055-4cb4-ba2a-54b02f8e90dd None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:40.625536 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe58d4bb-5052-46b3-9e21-d81c59fd7dc1 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:40.630024 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 326/664] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:40.635893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2ea592-6055-4cb4-ba2a-54b02f8e90dd None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:40.661401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:40.747345 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe58d4bb-5052-46b3-9e21-d81c59fd7dc1 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:40.750297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2ea592-6055-4cb4-ba2a-54b02f8e90dd None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:40.786780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f831564-9ecb-4b05-9b38-18b5e1453eed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:40.787714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b394d8f-dac5-4200-b39d-ce1813e7f924 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:40.788168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f831564-9ecb-4b05-9b38-18b5e1453eed None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:40.788343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b394d8f-dac5-4200-b39d-ce1813e7f924 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:40.788442 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f831564-9ecb-4b05-9b38-18b5e1453eed None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:40.788625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b394d8f-dac5-4200-b39d-ce1813e7f924 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:40.788724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f831564-9ecb-4b05-9b38-18b5e1453eed None admin] PATH_INFO: `/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/e0d0f8598fac40449bcb2a55ff8df8d9/roles/2e7fb4fa015442ec912f8d8760b86724` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:40.788900 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b394d8f-dac5-4200-b39d-ce1813e7f924 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:40.789479 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b394d8f-dac5-4200-b39d-ce1813e7f924 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:40.790842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b394d8f-dac5-4200-b39d-ce1813e7f924 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:40.790965 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f831564-9ecb-4b05-9b38-18b5e1453eed None admin] RBAC: Authorizing `identity:create_grant(project_id=4118e33b099f4cf0a3e3a46b211ba1d8, user_id=e0d0f8598fac40449bcb2a55ff8df8d9, role_id=2e7fb4fa015442ec912f8d8760b86724)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:40.792654 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f831564-9ecb-4b05-9b38-18b5e1453eed None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:40.801212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-5b394d8f-dac5-4200-b39d-ce1813e7f924 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 31 14:44:40.801212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:40.801212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:40.801212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:40.801212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:40.801212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:40.801212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:40.801212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:40.801212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 31 14:44:40.801212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:40.801212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:40.801212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:40.801212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:40.801212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:40.801212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:40.801212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:40.802192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:40.802192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:40.802192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:40.802192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:40.802192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:40.802192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:40.802192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:40.802192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:40.802192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:40.802192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:40.802192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:40.802192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:40.802192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:40.802192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:40.802192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:40.802192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:40.802192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:40.813147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:40.813147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:40.813147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:40.813147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:40.813147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:40.813147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:40.813147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:40.813147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:40.813147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:40.813147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:40.813147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:40.813147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:40.813147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 31 14:44:40.813147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:40.813147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 339/665] 10.222.0.36 () {56 vars in 1132 bytes} [Mon Oct 31 14:44:40 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:44:40.814032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe5d926-bb55-41fa-919d-6c58cd5cf38f None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:40.814032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe5d926-bb55-41fa-919d-6c58cd5cf38f None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:40.814032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe5d926-bb55-41fa-919d-6c58cd5cf38f None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:40.814204 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 327/666] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:44:40 2022] PUT /identity/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/e0d0f8598fac40449bcb2a55ff8df8d9/roles/2e7fb4fa015442ec912f8d8760b86724 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:44:40.814283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 340/667] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:40.814359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f831564-9ecb-4b05-9b38-18b5e1453eed None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:40.814428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f15345-e536-48f9-96c0-dbf2c19fa4e8 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:40.814428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f15345-e536-48f9-96c0-dbf2c19fa4e8 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:40.814428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f15345-e536-48f9-96c0-dbf2c19fa4e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:40.837978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-75f15345-e536-48f9-96c0-dbf2c19fa4e8 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:40.840146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-75f15345-e536-48f9-96c0-dbf2c19fa4e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:40.864901 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:40.928825 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:40.939714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 341/668] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:40 2022] POST /identity/v3/auth/tokens => generated 1373 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:40.960195 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f831564-9ecb-4b05-9b38-18b5e1453eed None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:41.000120 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c99282-7ae6-4078-aaaa-a55893a75a6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:41.000994 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c99282-7ae6-4078-aaaa-a55893a75a6a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:41.001277 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c99282-7ae6-4078-aaaa-a55893a75a6a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:41.001554 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c99282-7ae6-4078-aaaa-a55893a75a6a None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:41.002264 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c99282-7ae6-4078-aaaa-a55893a75a6a None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:41.002575 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c99282-7ae6-4078-aaaa-a55893a75a6a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:41.004030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c99282-7ae6-4078-aaaa-a55893a75a6a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:41.009313 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 328/669] 10.222.0.36 () {56 vars in 1126 bytes} [Mon Oct 31 14:44:40 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:41.016534 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f15345-e536-48f9-96c0-dbf2c19fa4e8 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:41.127015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f15345-e536-48f9-96c0-dbf2c19fa4e8 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:41.163614 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832e6b84-4399-4916-8002-29aebeba398c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:41.164229 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832e6b84-4399-4916-8002-29aebeba398c None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:41.164509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832e6b84-4399-4916-8002-29aebeba398c None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:41.164782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832e6b84-4399-4916-8002-29aebeba398c None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:41.165402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832e6b84-4399-4916-8002-29aebeba398c None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:41.165702 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832e6b84-4399-4916-8002-29aebeba398c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:41.167722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832e6b84-4399-4916-8002-29aebeba398c None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:41.173225 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 342/670] 10.222.0.36 () {56 vars in 1126 bytes} [Mon Oct 31 14:44:41 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:41.180411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c99282-7ae6-4078-aaaa-a55893a75a6a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:41.225732 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832e6b84-4399-4916-8002-29aebeba398c None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:41.247947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:41.311105 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c99282-7ae6-4078-aaaa-a55893a75a6a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:41.351707 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832e6b84-4399-4916-8002-29aebeba398c None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:41.358143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49afbd1-98b5-402d-9f5e-c187d1f10e48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:41.358763 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49afbd1-98b5-402d-9f5e-c187d1f10e48 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:41.359046 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49afbd1-98b5-402d-9f5e-c187d1f10e48 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:41.359324 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49afbd1-98b5-402d-9f5e-c187d1f10e48 None admin] PATH_INFO: `/v3/endpoints` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:41.360064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49afbd1-98b5-402d-9f5e-c187d1f10e48 None admin] RBAC: Adding query filter params (interface=public, service_id=f74a47526e404cbe8bd63a45adf7d56c, region_id=RegionOne) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:41.360921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49afbd1-98b5-402d-9f5e-c187d1f10e48 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:41.362307 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49afbd1-98b5-402d-9f5e-c187d1f10e48 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:41.370626 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 329/671] 10.222.0.36 () {56 vars in 1251 bytes} [Mon Oct 31 14:44:41 2022] GET /identity/v3/endpoints?service_id=f74a47526e404cbe8bd63a45adf7d56c&interface=public®ion_id=RegionOne => generated 194 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:41.375713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49afbd1-98b5-402d-9f5e-c187d1f10e48 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:41.391487 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:41.397031 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b22d61-79f0-440a-825d-27e1add710cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:41.397750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b22d61-79f0-440a-825d-27e1add710cb None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:41.398495 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b22d61-79f0-440a-825d-27e1add710cb None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:41.398973 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b22d61-79f0-440a-825d-27e1add710cb None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:41.400038 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b22d61-79f0-440a-825d-27e1add710cb None admin] RBAC: Authorizing `identity:create_service()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:41.401487 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b22d61-79f0-440a-825d-27e1add710cb None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:41.422269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 343/672] 10.222.0.36 () {60 vars in 1141 bytes} [Mon Oct 31 14:44:41 2022] POST /identity/v3/services => generated 251 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:44:41.501373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49afbd1-98b5-402d-9f5e-c187d1f10e48 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:41.553004 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e173e9b-5bed-42d5-bcd0-a41cd1c23db1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:41.553687 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e173e9b-5bed-42d5-bcd0-a41cd1c23db1 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:41.553973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e173e9b-5bed-42d5-bcd0-a41cd1c23db1 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:41.554249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e173e9b-5bed-42d5-bcd0-a41cd1c23db1 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:41.554880 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e173e9b-5bed-42d5-bcd0-a41cd1c23db1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:41.556332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e173e9b-5bed-42d5-bcd0-a41cd1c23db1 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:41.564556 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 330/673] 10.222.0.36 () {56 vars in 1087 bytes} [Mon Oct 31 14:44:41 2022] GET /identity/v3/services => generated 1497 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:44:42.166426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fa941d-1354-4e05-96a1-e735aa84c7e9 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:42.167025 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fa941d-1354-4e05-96a1-e735aa84c7e9 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:42.167468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fa941d-1354-4e05-96a1-e735aa84c7e9 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:42.169207 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 344/674] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:42.174886 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c34cdd-9ae0-410c-bb1f-c15c4d547cb9 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:42.175808 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c34cdd-9ae0-410c-bb1f-c15c4d547cb9 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:42.176292 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c34cdd-9ae0-410c-bb1f-c15c4d547cb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:42.195369 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-54c34cdd-9ae0-410c-bb1f-c15c4d547cb9 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:42.196904 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-54c34cdd-9ae0-410c-bb1f-c15c4d547cb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:42.263999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:42.275871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 331/675] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:42 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 31 14:44:42.540632 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fa941d-1354-4e05-96a1-e735aa84c7e9 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:42.621995 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:42.645784 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fa941d-1354-4e05-96a1-e735aa84c7e9 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:42.681055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfbd3f0-c7a5-47de-93cd-a8358c34ac5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:42.681685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfbd3f0-c7a5-47de-93cd-a8358c34ac5d None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:42.681962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfbd3f0-c7a5-47de-93cd-a8358c34ac5d None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:42.682234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfbd3f0-c7a5-47de-93cd-a8358c34ac5d None admin] PATH_INFO: `/v3/services/object-store` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:42.682961 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfbd3f0-c7a5-47de-93cd-a8358c34ac5d None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:42.684351 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfbd3f0-c7a5-47de-93cd-a8358c34ac5d None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:42.693284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-0bfbd3f0-c7a5-47de-93cd-a8358c34ac5d None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 31 14:44:42.693284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:42.693284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:42.693284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:42.693284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:42.693284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:42.693284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:42.693284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:42.693284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 31 14:44:42.693284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:42.693284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:42.693284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:42.693284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:42.693284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:42.693284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:42.693284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:42.693284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:42.694278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:42.694278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:42.694278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:42.694278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:42.694278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:42.694278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:42.694278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:42.694278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:42.694278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:42.694278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:42.694278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:42.694278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:42.694278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:42.694278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:42.694278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:42.694278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:42.695530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:42.695530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:42.695530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:42.695530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:42.695530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:42.695530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:42.695530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:42.695530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:42.695530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:42.695530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:42.695530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:42.695530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:42.695530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 31 14:44:42.695530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:42.695530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 345/676] 10.222.0.36 () {56 vars in 1126 bytes} [Mon Oct 31 14:44:42 2022] GET /identity/v3/services/object-store => generated 93 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:42.701101 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c34cdd-9ae0-410c-bb1f-c15c4d547cb9 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:42.810564 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c34cdd-9ae0-410c-bb1f-c15c4d547cb9 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:42.851209 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34394aa3-18b6-409f-bf22-cc710c212b58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:42.852197 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34394aa3-18b6-409f-bf22-cc710c212b58 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:42.852708 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34394aa3-18b6-409f-bf22-cc710c212b58 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:42.853165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34394aa3-18b6-409f-bf22-cc710c212b58 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:42.853993 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34394aa3-18b6-409f-bf22-cc710c212b58 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:42.854534 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34394aa3-18b6-409f-bf22-cc710c212b58 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:42.856137 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34394aa3-18b6-409f-bf22-cc710c212b58 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:42.862052 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 332/677] 10.222.0.36 () {56 vars in 1122 bytes} [Mon Oct 31 14:44:42 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:42.867928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfbd3f0-c7a5-47de-93cd-a8358c34ac5d None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:42.977731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfbd3f0-c7a5-47de-93cd-a8358c34ac5d None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:43.016194 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7669aa8-4521-49b1-ba81-d5d1d2f79878 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:43.016834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7669aa8-4521-49b1-ba81-d5d1d2f79878 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:43.017118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7669aa8-4521-49b1-ba81-d5d1d2f79878 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:43.017396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7669aa8-4521-49b1-ba81-d5d1d2f79878 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:43.018054 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7669aa8-4521-49b1-ba81-d5d1d2f79878 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:43.018358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7669aa8-4521-49b1-ba81-d5d1d2f79878 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:43.019804 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7669aa8-4521-49b1-ba81-d5d1d2f79878 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:43.026669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 346/678] 10.222.0.36 () {56 vars in 1122 bytes} [Mon Oct 31 14:44:42 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:43.032613 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34394aa3-18b6-409f-bf22-cc710c212b58 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:43.110598 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:43.143109 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34394aa3-18b6-409f-bf22-cc710c212b58 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:43.159478 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6170594-0b16-4ecc-b743-cefb260b35c8 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:43.159810 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6170594-0b16-4ecc-b743-cefb260b35c8 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:43.160060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6170594-0b16-4ecc-b743-cefb260b35c8 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:43.161647 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 347/679] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:43.167679 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66869b3f-958e-43bb-a05c-14042695a166 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:43.168223 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66869b3f-958e-43bb-a05c-14042695a166 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:43.168720 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66869b3f-958e-43bb-a05c-14042695a166 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:43.181796 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10a44d2-907c-4f68-b18d-8b1126baa10f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:43.182850 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:43.183079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10a44d2-907c-4f68-b18d-8b1126baa10f None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:43.183274 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10a44d2-907c-4f68-b18d-8b1126baa10f None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:43.185014 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10a44d2-907c-4f68-b18d-8b1126baa10f None admin] PATH_INFO: `/v3/endpoints` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:43.185014 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10a44d2-907c-4f68-b18d-8b1126baa10f None admin] RBAC: Adding query filter params (interface=public, service_id=d3ae3a35bd894e87b555d9135ffc42c6, region_id=RegionOne) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:43.185262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10a44d2-907c-4f68-b18d-8b1126baa10f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:43.186817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10a44d2-907c-4f68-b18d-8b1126baa10f None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:43.191872 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-66869b3f-958e-43bb-a05c-14042695a166 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:43.192608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 333/680] 10.222.0.36 () {56 vars in 1251 bytes} [Mon Oct 31 14:44:43 2022] GET /identity/v3/endpoints?service_id=d3ae3a35bd894e87b555d9135ffc42c6&interface=public®ion_id=RegionOne => generated 194 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:43.193483 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-66869b3f-958e-43bb-a05c-14042695a166 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:43.198062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10a44d2-907c-4f68-b18d-8b1126baa10f None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:43.275577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 348/681] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:43 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 31 14:44:43.312226 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10a44d2-907c-4f68-b18d-8b1126baa10f None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:43.340412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:43.350906 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bae5ea2-1ac4-4b78-b8c5-c5339d911c63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:43.351513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bae5ea2-1ac4-4b78-b8c5-c5339d911c63 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:43.351835 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bae5ea2-1ac4-4b78-b8c5-c5339d911c63 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:43.352108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bae5ea2-1ac4-4b78-b8c5-c5339d911c63 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:43.352713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bae5ea2-1ac4-4b78-b8c5-c5339d911c63 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:43.354698 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bae5ea2-1ac4-4b78-b8c5-c5339d911c63 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:43.362080 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 334/682] 10.222.0.36 () {56 vars in 1087 bytes} [Mon Oct 31 14:44:43 2022] GET /identity/v3/services => generated 1497 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:44:43.561690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66869b3f-958e-43bb-a05c-14042695a166 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:43.676442 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66869b3f-958e-43bb-a05c-14042695a166 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:43.714919 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b79a78a-94da-4d5f-90a5-470b0fe3d6c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:43.715582 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b79a78a-94da-4d5f-90a5-470b0fe3d6c6 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:43.715871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b79a78a-94da-4d5f-90a5-470b0fe3d6c6 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:43.716147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b79a78a-94da-4d5f-90a5-470b0fe3d6c6 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:43.719254 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:43.728156 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b79a78a-94da-4d5f-90a5-470b0fe3d6c6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:44:43.734098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-2b79a78a-94da-4d5f-90a5-470b0fe3d6c6 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 31 14:44:43.734098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:43.734098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:43.734098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:43.734098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:43.734098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:43.734098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:43.734098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:43.734098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:43.734098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:43.734098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:43.734098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:43.734098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:44:43.734098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:44:43.734098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:44:43.734098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:44:43.735150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:44:43.735150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:44:43.735150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:44:43.735150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:44:43.735150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:43.735150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:43.735150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:43.735150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:43.735150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:43.735150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:43.735150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:43.735150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:43.735150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:43.735150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:43.735150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:43.735150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:43.736053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:43.736053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:43.736053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:43.736053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:43.736053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:44:43.736053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:44:43.736053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:44:43.736053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:44:43.736053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:44:43.736053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:44:43.736053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 31 14:44:43.736053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:43.736053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 349/683] 10.222.0.36 () {56 vars in 1120 bytes} [Mon Oct 31 14:44:43 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:43.741911 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5ff6d0-2829-49fe-9a11-f66a16bfb4e1 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:43.742211 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5ff6d0-2829-49fe-9a11-f66a16bfb4e1 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:43.742454 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5ff6d0-2829-49fe-9a11-f66a16bfb4e1 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:43.744454 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b79a78a-94da-4d5f-90a5-470b0fe3d6c6 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:43.747892 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 335/684] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:43 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:43.758951 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb17d713-038f-4051-8239-96bcc37e2d97 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:43.758951 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb17d713-038f-4051-8239-96bcc37e2d97 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:43.758951 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb17d713-038f-4051-8239-96bcc37e2d97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:43.777099 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-bb17d713-038f-4051-8239-96bcc37e2d97 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:43.779166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-bb17d713-038f-4051-8239-96bcc37e2d97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:43.859897 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 336/685] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:43 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 31 14:44:43.862472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b79a78a-94da-4d5f-90a5-470b0fe3d6c6 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:43.867186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4f01e2-c7ec-4e7a-888e-ea577c97f31a None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:43.867474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4f01e2-c7ec-4e7a-888e-ea577c97f31a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:43.867787 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4f01e2-c7ec-4e7a-888e-ea577c97f31a None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:43.873968 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 337/686] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:44:43.875789 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de41af2b-5bce-4774-891c-a031b456c1b4 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:43.876075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de41af2b-5bce-4774-891c-a031b456c1b4 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:43.876349 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de41af2b-5bce-4774-891c-a031b456c1b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:43.899563 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-de41af2b-5bce-4774-891c-a031b456c1b4 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:43.900766 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955da612-cc5e-41cb-97d9-10eab1d8c718 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:43.900997 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-de41af2b-5bce-4774-891c-a031b456c1b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:43.901658 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955da612-cc5e-41cb-97d9-10eab1d8c718 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:43.901933 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955da612-cc5e-41cb-97d9-10eab1d8c718 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:43.902203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955da612-cc5e-41cb-97d9-10eab1d8c718 None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:43.903039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955da612-cc5e-41cb-97d9-10eab1d8c718 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:43.903342 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955da612-cc5e-41cb-97d9-10eab1d8c718 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:43.904917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955da612-cc5e-41cb-97d9-10eab1d8c718 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:43.913960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 350/687] 10.222.0.36 () {56 vars in 1115 bytes} [Mon Oct 31 14:44:43 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:43.920864 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955da612-cc5e-41cb-97d9-10eab1d8c718 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:43.991865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 338/688] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:43 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:44.007927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:44.042861 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955da612-cc5e-41cb-97d9-10eab1d8c718 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:44.077904 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d4b2b9-5fb3-4772-9878-ed33e08e6416 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:44.078525 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d4b2b9-5fb3-4772-9878-ed33e08e6416 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:44.078801 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d4b2b9-5fb3-4772-9878-ed33e08e6416 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:44.079070 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d4b2b9-5fb3-4772-9878-ed33e08e6416 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:44.090482 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d4b2b9-5fb3-4772-9878-ed33e08e6416 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:44.091960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d4b2b9-5fb3-4772-9878-ed33e08e6416 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:44.107396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-c7d4b2b9-5fb3-4772-9878-ed33e08e6416 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:44.107396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:44.107396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:44.107396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:44.107396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:44.107396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:44.107396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:44.107396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:44.107396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:44.107396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:44.107396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:44.107396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:44.107396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:44.107396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:44.107396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:44.107396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:44.108419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:44.108419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:44.108419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:44.108419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:44.108419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:44.108419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:44.108419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:44.108419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:44.108419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:44.108419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:44.108419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:44.108419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:44.108419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:44.108419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:44.108419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:44.108419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:44.108419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:44.109331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:44.109331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:44.109331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:44.109331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 351/689] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:44.114465 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de41af2b-5bce-4774-891c-a031b456c1b4 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:44.163179 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d4b2b9-5fb3-4772-9878-ed33e08e6416 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:44.172848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:44.238759 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de41af2b-5bce-4774-891c-a031b456c1b4 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:44.289649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f7debe-b1dc-4e6a-be80-0961fdf44b64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:44.290192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f7debe-b1dc-4e6a-be80-0961fdf44b64 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:44.290474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f7debe-b1dc-4e6a-be80-0961fdf44b64 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:44.290747 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f7debe-b1dc-4e6a-be80-0961fdf44b64 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:44.290960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d4b2b9-5fb3-4772-9878-ed33e08e6416 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:44.291399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f7debe-b1dc-4e6a-be80-0961fdf44b64 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:44.292434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f7debe-b1dc-4e6a-be80-0961fdf44b64 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:44.295187 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f7debe-b1dc-4e6a-be80-0961fdf44b64 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:44.303271 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:44.311431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 339/690] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:44.312712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f7debe-b1dc-4e6a-be80-0961fdf44b64 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:44.334894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510b0b5f-253d-48dd-9f0d-1c3f1020505a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:44.335526 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510b0b5f-253d-48dd-9f0d-1c3f1020505a None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:44.335834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510b0b5f-253d-48dd-9f0d-1c3f1020505a None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:44.336113 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510b0b5f-253d-48dd-9f0d-1c3f1020505a None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:44.336692 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510b0b5f-253d-48dd-9f0d-1c3f1020505a None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:44.338050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510b0b5f-253d-48dd-9f0d-1c3f1020505a None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:44.348975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-510b0b5f-253d-48dd-9f0d-1c3f1020505a None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 31 14:44:44.348975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:44.348975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:44.348975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:44.348975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:44.348975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:44.348975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:44.348975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:44.348975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 31 14:44:44.348975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:44.348975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:44.348975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:44.348975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:44.348975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:44.348975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:44.348975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:44.348975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:44.349987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:44.349987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:44.349987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:44.349987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:44.349987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:44.349987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:44.349987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:44.349987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:44.349987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:44.349987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:44.349987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:44.349987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:44.349987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:44.349987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:44.349987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:44.349987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:44.350813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:44.350813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:44.350813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:44.350813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:44.350813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:44.350813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:44.350813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:44.350813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:44.350813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:44.350813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:44.350813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:44.350813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:44.350813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 31 14:44:44.350813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:44.350813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 352/691] 10.222.0.36 () {56 vars in 1114 bytes} [Mon Oct 31 14:44:44 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:44:44.351629 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510b0b5f-253d-48dd-9f0d-1c3f1020505a None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:44.420497 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f7debe-b1dc-4e6a-be80-0961fdf44b64 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:44.431128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:44.457471 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8256aca0-786f-4562-bbbd-ae480f35f49e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:44.458392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8256aca0-786f-4562-bbbd-ae480f35f49e None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:44.458672 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8256aca0-786f-4562-bbbd-ae480f35f49e None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:44.458940 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8256aca0-786f-4562-bbbd-ae480f35f49e None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:44.459510 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8256aca0-786f-4562-bbbd-ae480f35f49e None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:44.459634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510b0b5f-253d-48dd-9f0d-1c3f1020505a None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:44.460894 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8256aca0-786f-4562-bbbd-ae480f35f49e None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:44.470794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-8256aca0-786f-4562-bbbd-ae480f35f49e None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 31 14:44:44.470794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:44.470794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:44.470794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:44.470794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:44.470794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:44.470794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:44.470794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:44.470794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 31 14:44:44.470794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:44.470794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:44.470794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:44.470794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:44.470794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:44.470794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:44.470794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:44.471811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:44.471811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:44.471811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:44.471811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:44.471811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:44.471811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:44.471811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:44.471811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:44.471811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:44.471811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:44.471811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:44.471811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:44.471811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:44.471811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:44.471811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:44.471811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:44.471811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:44.473140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:44.473140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:44.473140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:44.473140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:44.473140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:44.473140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:44.473140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:44.473140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:44.473140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:44.473140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:44.473140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:44.473140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:44.473140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 31 14:44:44.473140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:44.476735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 340/692] 10.222.0.36 () {56 vars in 1132 bytes} [Mon Oct 31 14:44:44 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:44.477557 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8256aca0-786f-4562-bbbd-ae480f35f49e None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:44.503058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1113323-a922-41ed-a446-2aedb23945a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:44.504360 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1113323-a922-41ed-a446-2aedb23945a6 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:44.505077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1113323-a922-41ed-a446-2aedb23945a6 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:44.505512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1113323-a922-41ed-a446-2aedb23945a6 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:44.514946 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1113323-a922-41ed-a446-2aedb23945a6 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:44.516608 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1113323-a922-41ed-a446-2aedb23945a6 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:44.531763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-c1113323-a922-41ed-a446-2aedb23945a6 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 31 14:44:44.531763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:44.531763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:44.531763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:44.531763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:44.531763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:44.531763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:44.531763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:44.531763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:44.531763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:44.531763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:44.531763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:44.531763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:44.531763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:44.531763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:44.531763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:44.532758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:44.533684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:44.533684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:44.533684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:44.533684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:44.533684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:44.533684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:44.533684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:44.533684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:44.533684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 31 14:44:44.533684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:44.533684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 353/693] 10.222.0.36 () {56 vars in 1117 bytes} [Mon Oct 31 14:44:44 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:44.534947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1113323-a922-41ed-a446-2aedb23945a6 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:44.577232 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:44.592057 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8256aca0-786f-4562-bbbd-ae480f35f49e None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:44.630531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fa9a13-9f8f-4ded-a34c-201200940a18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:44.631191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fa9a13-9f8f-4ded-a34c-201200940a18 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:44.631475 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fa9a13-9f8f-4ded-a34c-201200940a18 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:44.631767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fa9a13-9f8f-4ded-a34c-201200940a18 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:44.632424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43fa9a13-9f8f-4ded-a34c-201200940a18 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:44.632725 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43fa9a13-9f8f-4ded-a34c-201200940a18 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:44.634090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43fa9a13-9f8f-4ded-a34c-201200940a18 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:44.639638 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 341/694] 10.222.0.36 () {56 vars in 1114 bytes} [Mon Oct 31 14:44:44 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:44.640289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fa9a13-9f8f-4ded-a34c-201200940a18 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:44.655456 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1113323-a922-41ed-a446-2aedb23945a6 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:44.684274 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:44.693074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f8eb57-18c4-4e3b-be9e-09535266f7c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:44.693681 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f8eb57-18c4-4e3b-be9e-09535266f7c7 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:44.694026 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f8eb57-18c4-4e3b-be9e-09535266f7c7 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:44.694258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f8eb57-18c4-4e3b-be9e-09535266f7c7 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:44.694894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f8eb57-18c4-4e3b-be9e-09535266f7c7 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:44.695281 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f8eb57-18c4-4e3b-be9e-09535266f7c7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:44.696648 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f8eb57-18c4-4e3b-be9e-09535266f7c7 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:44.706910 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 354/695] 10.222.0.36 () {56 vars in 1126 bytes} [Mon Oct 31 14:44:44 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:44.706910 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f8eb57-18c4-4e3b-be9e-09535266f7c7 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:44.753861 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fa9a13-9f8f-4ded-a34c-201200940a18 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:44.789732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6d929e-e475-4ba4-b247-0af4a64ebf3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:44.790332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6d929e-e475-4ba4-b247-0af4a64ebf3c None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:44.790606 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6d929e-e475-4ba4-b247-0af4a64ebf3c None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:44.790874 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6d929e-e475-4ba4-b247-0af4a64ebf3c None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:44.792221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6d929e-e475-4ba4-b247-0af4a64ebf3c None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:44.792530 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6d929e-e475-4ba4-b247-0af4a64ebf3c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:44.793954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6d929e-e475-4ba4-b247-0af4a64ebf3c None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:44.810908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:44.812077 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 342/696] 10.222.0.36 () {56 vars in 1149 bytes} [Mon Oct 31 14:44:44 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:44.812870 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6d929e-e475-4ba4-b247-0af4a64ebf3c None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:44.819616 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f8eb57-18c4-4e3b-be9e-09535266f7c7 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:44.857124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a439d81e-7fe2-4822-baec-7cf7474819c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:44.857654 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a439d81e-7fe2-4822-baec-7cf7474819c7 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:44.857928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a439d81e-7fe2-4822-baec-7cf7474819c7 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:44.858196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a439d81e-7fe2-4822-baec-7cf7474819c7 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:44.858811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a439d81e-7fe2-4822-baec-7cf7474819c7 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:44.859106 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a439d81e-7fe2-4822-baec-7cf7474819c7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:44.860539 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a439d81e-7fe2-4822-baec-7cf7474819c7 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:44.870933 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 355/697] 10.222.0.36 () {56 vars in 1114 bytes} [Mon Oct 31 14:44:44 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:44.870933 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a439d81e-7fe2-4822-baec-7cf7474819c7 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:44.918793 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:44.925974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6d929e-e475-4ba4-b247-0af4a64ebf3c None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:44.961334 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b58469d-9884-469d-9c06-f815be972952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:44.961955 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b58469d-9884-469d-9c06-f815be972952 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:44.962239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b58469d-9884-469d-9c06-f815be972952 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:44.962516 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b58469d-9884-469d-9c06-f815be972952 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:44.963147 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b58469d-9884-469d-9c06-f815be972952 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:44.963579 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b58469d-9884-469d-9c06-f815be972952 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:44.965025 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b58469d-9884-469d-9c06-f815be972952 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:44.970124 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 343/698] 10.222.0.36 () {56 vars in 1126 bytes} [Mon Oct 31 14:44:44 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:44.970910 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b58469d-9884-469d-9c06-f815be972952 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:44.983955 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a439d81e-7fe2-4822-baec-7cf7474819c7 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:45.019529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a6fb27-f315-4df2-92e5-787e6397f65e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:45.020629 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a6fb27-f315-4df2-92e5-787e6397f65e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:45.021004 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a6fb27-f315-4df2-92e5-787e6397f65e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:45.021366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a6fb27-f315-4df2-92e5-787e6397f65e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:45.057869 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:45.083834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b58469d-9884-469d-9c06-f815be972952 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:45.098066 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a6fb27-f315-4df2-92e5-787e6397f65e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:45.099508 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a6fb27-f315-4df2-92e5-787e6397f65e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:45.120212 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62feecb-6931-4c95-b4b1-23988c01985a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:45.120739 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62feecb-6931-4c95-b4b1-23988c01985a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:45.121013 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62feecb-6931-4c95-b4b1-23988c01985a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:45.121283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62feecb-6931-4c95-b4b1-23988c01985a None admin] PATH_INFO: `/v3/endpoints` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:45.121978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62feecb-6931-4c95-b4b1-23988c01985a None admin] RBAC: Adding query filter params (interface=public, service_id=60a50eed0b024a61afc006806a37eb25, region_id=RegionOne) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:45.122268 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62feecb-6931-4c95-b4b1-23988c01985a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:45.123650 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62feecb-6931-4c95-b4b1-23988c01985a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:45.133424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 344/699] 10.222.0.36 () {56 vars in 1251 bytes} [Mon Oct 31 14:44:44 2022] GET /identity/v3/endpoints?service_id=60a50eed0b024a61afc006806a37eb25&interface=public®ion_id=RegionOne => generated 194 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:45.133424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62feecb-6931-4c95-b4b1-23988c01985a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:45.157207 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:45.217508 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 356/700] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:44:44 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 31 14:44:45.217508 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a6fb27-f315-4df2-92e5-787e6397f65e None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:45.236885 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62feecb-6931-4c95-b4b1-23988c01985a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:45.269995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a729ea0a-3c29-4f4f-8cea-adf2804b9a5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:45.270649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a729ea0a-3c29-4f4f-8cea-adf2804b9a5a None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:45.270861 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a729ea0a-3c29-4f4f-8cea-adf2804b9a5a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:45.271140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a729ea0a-3c29-4f4f-8cea-adf2804b9a5a None admin] PATH_INFO: `/v3/endpoints` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:45.271829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a729ea0a-3c29-4f4f-8cea-adf2804b9a5a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:45.273187 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a729ea0a-3c29-4f4f-8cea-adf2804b9a5a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:45.284800 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:45.296480 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 345/701] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:44:45 2022] POST /identity/v3/endpoints => generated 347 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:45.297387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a729ea0a-3c29-4f4f-8cea-adf2804b9a5a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:45.350866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a6fb27-f315-4df2-92e5-787e6397f65e None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:45.406080 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:45.408102 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c5b17f-8206-46b8-bae9-d0f91019e483 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:45.409022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c5b17f-8206-46b8-bae9-d0f91019e483 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:45.409483 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c5b17f-8206-46b8-bae9-d0f91019e483 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:45.409933 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c5b17f-8206-46b8-bae9-d0f91019e483 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:45.410979 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c5b17f-8206-46b8-bae9-d0f91019e483 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:45.414070 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c5b17f-8206-46b8-bae9-d0f91019e483 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:45.423584 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 357/702] 10.222.0.36 () {56 vars in 1087 bytes} [Mon Oct 31 14:44:45 2022] GET /identity/v3/services => generated 1497 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:44:45.441598 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a729ea0a-3c29-4f4f-8cea-adf2804b9a5a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:45.489188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1526fb50-540c-4771-a97a-bf3fa8fb6f55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:45.490260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1526fb50-540c-4771-a97a-bf3fa8fb6f55 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:45.490736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1526fb50-540c-4771-a97a-bf3fa8fb6f55 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:45.491197 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1526fb50-540c-4771-a97a-bf3fa8fb6f55 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:45.505220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1526fb50-540c-4771-a97a-bf3fa8fb6f55 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:45.507046 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1526fb50-540c-4771-a97a-bf3fa8fb6f55 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:45.524249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-1526fb50-540c-4771-a97a-bf3fa8fb6f55 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:45.524249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:45.524249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:45.524249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:45.524249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:45.524249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:45.524249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:45.524249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:45.524249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:45.524249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:45.524249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:45.524249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:45.524249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:44:45.524249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:44:45.524249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:44:45.524249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:44:45.525241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:45.525241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:45.525241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:45.525241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:45.525241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:45.525241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:45.525241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:45.525241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:45.525241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:45.525241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:45.525241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:45.525241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:45.525241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:45.525241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:45.525241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:45.525241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:45.525241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:44:45.526755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:44:45.526755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:44:45.526755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:45.529073 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 346/703] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:45.535018 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c5b17f-8206-46b8-bae9-d0f91019e483 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:45.601462 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8630a74-a8e0-4b68-9227-1105ccaeddb2 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:45.601795 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8630a74-a8e0-4b68-9227-1105ccaeddb2 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:45.602044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8630a74-a8e0-4b68-9227-1105ccaeddb2 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:45.603756 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 347/704] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:45.611402 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28738fc-fcbc-46c0-a2fa-777b5b53de3d None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:45.611835 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28738fc-fcbc-46c0-a2fa-777b5b53de3d None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:45.612203 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28738fc-fcbc-46c0-a2fa-777b5b53de3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:45.638919 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-a28738fc-fcbc-46c0-a2fa-777b5b53de3d None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:45.640857 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a28738fc-fcbc-46c0-a2fa-777b5b53de3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:45.663409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c5b17f-8206-46b8-bae9-d0f91019e483 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:45.691383 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:45.707081 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee144a6b-7aa4-4712-b211-0d605b3d6b9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:45.707734 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee144a6b-7aa4-4712-b211-0d605b3d6b9e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:45.708017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee144a6b-7aa4-4712-b211-0d605b3d6b9e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:45.708295 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee144a6b-7aa4-4712-b211-0d605b3d6b9e None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:45.709223 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee144a6b-7aa4-4712-b211-0d605b3d6b9e None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:45.709531 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee144a6b-7aa4-4712-b211-0d605b3d6b9e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:45.710899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee144a6b-7aa4-4712-b211-0d605b3d6b9e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:45.724427 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 358/705] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:44:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:45.730140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee144a6b-7aa4-4712-b211-0d605b3d6b9e None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:45.740241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 348/706] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:45 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 31 14:44:45.836590 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee144a6b-7aa4-4712-b211-0d605b3d6b9e None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:45.872036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7d93b3-05e4-4604-8759-abf725a613a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:45.872662 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7d93b3-05e4-4604-8759-abf725a613a0 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:45.872939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7d93b3-05e4-4604-8759-abf725a613a0 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:45.873211 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7d93b3-05e4-4604-8759-abf725a613a0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:45.880116 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7d93b3-05e4-4604-8759-abf725a613a0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:45.881549 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7d93b3-05e4-4604-8759-abf725a613a0 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:45.892996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-7a7d93b3-05e4-4604-8759-abf725a613a0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:45.892996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:45.892996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:45.892996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:45.892996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:45.892996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:45.892996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:45.892996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:45.892996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:45.892996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:45.892996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:45.892996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:45.892996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:45.892996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:45.892996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:45.893962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:45.893962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:45.893962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:45.893962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:45.893962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:45.893962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:45.893962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:45.893962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:45.893962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:45.893962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:45.893962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:45.893962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:45.893962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:45.893962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:45.893962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:45.893962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:45.893962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:45.894923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:45.894923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:45.894923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:45.894923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:45.894923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:45.894923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:45.894923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:45.894923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:45.894923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 359/707] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:44:45.902266 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28738fc-fcbc-46c0-a2fa-777b5b53de3d None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:45.972899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:46.013892 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28738fc-fcbc-46c0-a2fa-777b5b53de3d None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:46.048102 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7d93b3-05e4-4604-8759-abf725a613a0 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:46.050069 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ca5893-2333-411a-bb39-b0ab1ed6a0f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:46.050678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ca5893-2333-411a-bb39-b0ab1ed6a0f6 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:46.050958 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ca5893-2333-411a-bb39-b0ab1ed6a0f6 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:46.051229 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ca5893-2333-411a-bb39-b0ab1ed6a0f6 None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:46.051950 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ca5893-2333-411a-bb39-b0ab1ed6a0f6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:46.052340 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ca5893-2333-411a-bb39-b0ab1ed6a0f6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:46.053765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ca5893-2333-411a-bb39-b0ab1ed6a0f6 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:46.066371 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 349/708] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:44:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:46.072150 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ca5893-2333-411a-bb39-b0ab1ed6a0f6 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:46.103129 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:46.169956 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7d93b3-05e4-4604-8759-abf725a613a0 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:46.212127 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cfe8f4-6573-42d8-86f9-b9c4450af977 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:46.213114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cfe8f4-6573-42d8-86f9-b9c4450af977 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:46.213619 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cfe8f4-6573-42d8-86f9-b9c4450af977 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:46.213736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ca5893-2333-411a-bb39-b0ab1ed6a0f6 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:46.214259 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cfe8f4-6573-42d8-86f9-b9c4450af977 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:46.215033 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cfe8f4-6573-42d8-86f9-b9c4450af977 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:46.216644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cfe8f4-6573-42d8-86f9-b9c4450af977 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:46.226400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-c1cfe8f4-6573-42d8-86f9-b9c4450af977 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 31 14:44:46.226400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:46.226400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:46.226400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:46.226400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:46.226400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:46.226400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:46.226400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:46.226400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 31 14:44:46.226400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:46.226400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:46.226400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:46.226400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:46.226400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:46.226400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:46.226400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:46.226400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:46.227423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:46.227423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:46.227423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:46.227423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:46.227423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:46.227423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:46.227423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:46.227423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:46.227423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:46.227423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:46.227423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:46.227423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:46.227423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:46.227423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:46.227423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:46.227423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:46.228287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:46.228287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:46.228287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:46.228287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:46.228287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:46.228287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:46.228287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:46.228287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:46.228287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:46.228287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:46.228287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:46.228287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:46.228287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 31 14:44:46.228287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:46.235912 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 360/709] 10.222.0.36 () {56 vars in 1126 bytes} [Mon Oct 31 14:44:46 2022] GET /identity/v3/services/object-store => generated 93 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:46.237833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cfe8f4-6573-42d8-86f9-b9c4450af977 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:46.240007 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:46.256473 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9095537-9ea9-4189-ba70-82ca43fc6151 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:46.257080 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9095537-9ea9-4189-ba70-82ca43fc6151 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:46.257357 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9095537-9ea9-4189-ba70-82ca43fc6151 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:46.257625 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9095537-9ea9-4189-ba70-82ca43fc6151 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:46.258615 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9095537-9ea9-4189-ba70-82ca43fc6151 None admin] RBAC: Adding query filter params (role.id=2e7fb4fa015442ec912f8d8760b86724, scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8, user.id=e0d0f8598fac40449bcb2a55ff8df8d9) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:46.258921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9095537-9ea9-4189-ba70-82ca43fc6151 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:46.261311 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9095537-9ea9-4189-ba70-82ca43fc6151 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:46.271634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 350/710] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:44:46 2022] GET /identity/v3/role_assignments?user.id=e0d0f8598fac40449bcb2a55ff8df8d9&scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8&role.id=2e7fb4fa015442ec912f8d8760b86724 => generated 608 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:46.369275 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cfe8f4-6573-42d8-86f9-b9c4450af977 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:46.378388 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:46.410700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b9f846-7364-480e-a897-5b5abebc05f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:46.411796 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b9f846-7364-480e-a897-5b5abebc05f4 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:46.412304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b9f846-7364-480e-a897-5b5abebc05f4 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:46.412772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b9f846-7364-480e-a897-5b5abebc05f4 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:46.413626 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b9f846-7364-480e-a897-5b5abebc05f4 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:46.414133 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b9f846-7364-480e-a897-5b5abebc05f4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:46.415858 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b9f846-7364-480e-a897-5b5abebc05f4 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:46.422592 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 361/711] 10.222.0.36 () {56 vars in 1122 bytes} [Mon Oct 31 14:44:46 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:46.428703 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9095537-9ea9-4189-ba70-82ca43fc6151 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:46.552837 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9095537-9ea9-4189-ba70-82ca43fc6151 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:46.593769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807a6eba-d797-43aa-a263-ca60e7260b25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:46.594668 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807a6eba-d797-43aa-a263-ca60e7260b25 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:46.595124 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807a6eba-d797-43aa-a263-ca60e7260b25 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:46.595619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807a6eba-d797-43aa-a263-ca60e7260b25 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:46.596411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807a6eba-d797-43aa-a263-ca60e7260b25 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:46.596909 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807a6eba-d797-43aa-a263-ca60e7260b25 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:46.598435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807a6eba-d797-43aa-a263-ca60e7260b25 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:46.604022 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 351/712] 10.222.0.36 () {56 vars in 1122 bytes} [Mon Oct 31 14:44:46 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:46.609525 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b9f846-7364-480e-a897-5b5abebc05f4 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:46.712261 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b9f846-7364-480e-a897-5b5abebc05f4 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:46.748045 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c1af02-f7d3-42cf-a3ed-f91bf3474c26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:46.749033 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c1af02-f7d3-42cf-a3ed-f91bf3474c26 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:46.749511 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c1af02-f7d3-42cf-a3ed-f91bf3474c26 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:46.750002 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c1af02-f7d3-42cf-a3ed-f91bf3474c26 None admin] PATH_INFO: `/v3/endpoints` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:46.750866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c1af02-f7d3-42cf-a3ed-f91bf3474c26 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:46.752466 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c1af02-f7d3-42cf-a3ed-f91bf3474c26 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:46.774883 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 362/713] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:44:46 2022] POST /identity/v3/endpoints => generated 349 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:47.628866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf07ab5-dd0d-4d0a-96c5-d5781adcf986 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:47.629388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf07ab5-dd0d-4d0a-96c5-d5781adcf986 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:47.629388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf07ab5-dd0d-4d0a-96c5-d5781adcf986 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:47.631058 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 352/714] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:47.637753 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aad7b01-d516-4c66-8e67-f1405f34c3f6 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:47.638071 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aad7b01-d516-4c66-8e67-f1405f34c3f6 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:47.638309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aad7b01-d516-4c66-8e67-f1405f34c3f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:47.657902 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-7aad7b01-d516-4c66-8e67-f1405f34c3f6 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:47.659126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7aad7b01-d516-4c66-8e67-f1405f34c3f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:47.681522 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:47.738585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 363/715] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:47 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:47.770949 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27777f4f-6d6d-4ba2-91a0-5c07d1df3629 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:47.771606 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27777f4f-6d6d-4ba2-91a0-5c07d1df3629 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:47.772105 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27777f4f-6d6d-4ba2-91a0-5c07d1df3629 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:47.773929 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 353/716] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:47.780808 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10a0ef2-83e9-47c8-ac2b-4c947424f66f None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:47.781339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10a0ef2-83e9-47c8-ac2b-4c947424f66f None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:47.781814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10a0ef2-83e9-47c8-ac2b-4c947424f66f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:47.801431 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-e10a0ef2-83e9-47c8-ac2b-4c947424f66f None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:47.802875 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e10a0ef2-83e9-47c8-ac2b-4c947424f66f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:47.874920 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 364/717] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:47 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:48.007827 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27777f4f-6d6d-4ba2-91a0-5c07d1df3629 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:48.067037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:48.122123 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27777f4f-6d6d-4ba2-91a0-5c07d1df3629 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:48.148439 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10a0ef2-83e9-47c8-ac2b-4c947424f66f None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:48.161358 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a456687-efd8-4045-8551-ce8dab405cd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:48.161987 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a456687-efd8-4045-8551-ce8dab405cd6 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:48.162265 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a456687-efd8-4045-8551-ce8dab405cd6 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:48.162533 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a456687-efd8-4045-8551-ce8dab405cd6 None admin] PATH_INFO: `/v3/services/compute` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:48.163108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a456687-efd8-4045-8551-ce8dab405cd6 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:48.164585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a456687-efd8-4045-8551-ce8dab405cd6 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:48.174983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-5a456687-efd8-4045-8551-ce8dab405cd6 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 31 14:44:48.174983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:48.174983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:48.174983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:48.174983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:48.174983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:48.174983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:48.174983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:48.174983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 31 14:44:48.174983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:48.174983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:48.174983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:48.174983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:48.174983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:48.174983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:48.174983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:48.174983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:48.176354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:48.176354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:48.176354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:48.176354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:48.176354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:48.176354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:48.176354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:48.176354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:48.176354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:48.176354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:48.176354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:48.176354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:48.176354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:48.176354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:48.176354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:48.176354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:48.177577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:48.177577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:48.177577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:48.177577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:48.177577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:48.177577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:48.177577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:48.177577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:48.177577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:48.177577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:48.177577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:48.177577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:48.177577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 31 14:44:48.177577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:48.179798 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 354/718] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:48 2022] GET /identity/v3/services/compute => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:48.182937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a456687-efd8-4045-8551-ce8dab405cd6 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:48.257039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:48.282469 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10a0ef2-83e9-47c8-ac2b-4c947424f66f None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:48.305177 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a456687-efd8-4045-8551-ce8dab405cd6 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:48.318792 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02637ae5-7972-4a32-bf52-5040c5cf9584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:48.319213 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02637ae5-7972-4a32-bf52-5040c5cf9584 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:48.319494 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02637ae5-7972-4a32-bf52-5040c5cf9584 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:48.319817 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02637ae5-7972-4a32-bf52-5040c5cf9584 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:48.320400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02637ae5-7972-4a32-bf52-5040c5cf9584 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:48.321757 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02637ae5-7972-4a32-bf52-5040c5cf9584 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:48.326662 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:48.331937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-02637ae5-7972-4a32-bf52-5040c5cf9584 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 31 14:44:48.331937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:48.331937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:48.331937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:48.331937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:48.331937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:48.331937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:48.331937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:48.331937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 31 14:44:48.331937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:48.331937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:48.331937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:48.331937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:48.331937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:48.331937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:48.331937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:48.331937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:48.332908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:48.332908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:48.332908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:48.332908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:48.332908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:48.332908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:48.332908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:48.332908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:48.332908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:48.332908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:48.332908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:48.332908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:48.332908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:48.332908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:48.332908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:48.332908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:48.333751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:48.333751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:48.333751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:48.333751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:48.333751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:48.333751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:48.333751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:48.333751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:48.333751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:48.333751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:48.333751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:48.333751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:48.333751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 31 14:44:48.333751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:48.338237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 365/719] 10.222.0.36 () {56 vars in 1114 bytes} [Mon Oct 31 14:44:48 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:48.339211 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02637ae5-7972-4a32-bf52-5040c5cf9584 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:48.354777 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5257cf1b-9e27-4521-b2a7-4ba7d143703f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:48.355458 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5257cf1b-9e27-4521-b2a7-4ba7d143703f None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:48.355741 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5257cf1b-9e27-4521-b2a7-4ba7d143703f None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:48.355956 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5257cf1b-9e27-4521-b2a7-4ba7d143703f None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:48.356580 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5257cf1b-9e27-4521-b2a7-4ba7d143703f None admin] RBAC: Adding query filter params (name=compute) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:48.357037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5257cf1b-9e27-4521-b2a7-4ba7d143703f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:48.359374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5257cf1b-9e27-4521-b2a7-4ba7d143703f None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:48.369940 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 355/720] 10.222.0.36 () {56 vars in 1112 bytes} [Mon Oct 31 14:44:48 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:48.370928 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5257cf1b-9e27-4521-b2a7-4ba7d143703f None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:48.468636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02637ae5-7972-4a32-bf52-5040c5cf9584 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:48.491165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5257cf1b-9e27-4521-b2a7-4ba7d143703f None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:48.509740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c0771d-78ac-4365-b52a-47de8a35d082 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:48.510800 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c0771d-78ac-4365-b52a-47de8a35d082 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:48.511360 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c0771d-78ac-4365-b52a-47de8a35d082 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:48.511867 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c0771d-78ac-4365-b52a-47de8a35d082 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:48.512707 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c0771d-78ac-4365-b52a-47de8a35d082 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:48.513216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c0771d-78ac-4365-b52a-47de8a35d082 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:48.514796 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c0771d-78ac-4365-b52a-47de8a35d082 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:48.525642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 366/721] 10.222.0.36 () {56 vars in 1114 bytes} [Mon Oct 31 14:44:48 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:48.527595 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c0771d-78ac-4365-b52a-47de8a35d082 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:48.532715 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a89b9e-a3bf-4646-8d5a-b7cae653201f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:48.533331 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a89b9e-a3bf-4646-8d5a-b7cae653201f None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:48.533618 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a89b9e-a3bf-4646-8d5a-b7cae653201f None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:48.533892 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a89b9e-a3bf-4646-8d5a-b7cae653201f None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:48.534531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a89b9e-a3bf-4646-8d5a-b7cae653201f None admin] RBAC: Adding query filter params (type=compute) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:48.534837 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a89b9e-a3bf-4646-8d5a-b7cae653201f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:48.536312 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a89b9e-a3bf-4646-8d5a-b7cae653201f None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:48.551967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:48.552205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 356/722] 10.222.0.36 () {56 vars in 1112 bytes} [Mon Oct 31 14:44:48 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:48.572065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84e6675-454f-4760-aa8a-67b06162cd4f None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:48.572257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84e6675-454f-4760-aa8a-67b06162cd4f None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:48.572560 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84e6675-454f-4760-aa8a-67b06162cd4f None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:48.575091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 357/723] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:44:48.580813 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48b5512-5b49-4e4f-a58b-d76294354658 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:48.581134 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48b5512-5b49-4e4f-a58b-d76294354658 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:48.581371 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48b5512-5b49-4e4f-a58b-d76294354658 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:48.602570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-f48b5512-5b49-4e4f-a58b-d76294354658 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:48.604694 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-f48b5512-5b49-4e4f-a58b-d76294354658 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:48.649404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:48.653129 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c0771d-78ac-4365-b52a-47de8a35d082 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:48.694661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d15378-49ad-433e-8258-4c532a4f3277 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:48.695391 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d15378-49ad-433e-8258-4c532a4f3277 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:48.695602 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 358/724] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:48 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:48.695802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d15378-49ad-433e-8258-4c532a4f3277 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:48.696129 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d15378-49ad-433e-8258-4c532a4f3277 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:48.696771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d15378-49ad-433e-8258-4c532a4f3277 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:48.697143 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d15378-49ad-433e-8258-4c532a4f3277 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:48.698596 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d15378-49ad-433e-8258-4c532a4f3277 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:48.705835 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 367/725] 10.222.0.36 () {56 vars in 1114 bytes} [Mon Oct 31 14:44:48 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:48.710191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48b5512-5b49-4e4f-a58b-d76294354658 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:48.821580 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48b5512-5b49-4e4f-a58b-d76294354658 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:48.864364 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf5458f-33ce-4a2d-8b6a-e89a3ef23a5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:48.864938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf5458f-33ce-4a2d-8b6a-e89a3ef23a5a None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:48.865219 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf5458f-33ce-4a2d-8b6a-e89a3ef23a5a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:48.867600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf5458f-33ce-4a2d-8b6a-e89a3ef23a5a None admin] PATH_INFO: `/v3/endpoints` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:48.867600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf5458f-33ce-4a2d-8b6a-e89a3ef23a5a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:48.867600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf5458f-33ce-4a2d-8b6a-e89a3ef23a5a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:48.906670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 359/726] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:44:48 2022] POST /identity/v3/endpoints => generated 346 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:49.002554 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d15378-49ad-433e-8258-4c532a4f3277 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:49.104870 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f7c3ac-8833-46e6-8822-9355ee9bb417 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:49.105153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f7c3ac-8833-46e6-8822-9355ee9bb417 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:49.105420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f7c3ac-8833-46e6-8822-9355ee9bb417 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:49.107033 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 360/727] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:49.112692 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad32e61-14ea-4c61-bf22-cc966bd45c1d None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:49.112907 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:49.112992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad32e61-14ea-4c61-bf22-cc966bd45c1d None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:49.113208 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad32e61-14ea-4c61-bf22-cc966bd45c1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:49.137675 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-4ad32e61-14ea-4c61-bf22-cc966bd45c1d None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:49.141844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-4ad32e61-14ea-4c61-bf22-cc966bd45c1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:49.161645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d15378-49ad-433e-8258-4c532a4f3277 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:49.183041 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:49.224795 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0d9871-3dd7-4862-aee9-59c6217ae2fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:49.225434 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0d9871-3dd7-4862-aee9-59c6217ae2fe None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:49.225715 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0d9871-3dd7-4862-aee9-59c6217ae2fe None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:49.227584 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0d9871-3dd7-4862-aee9-59c6217ae2fe None admin] PATH_INFO: `/v3/services/image` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:49.227584 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0d9871-3dd7-4862-aee9-59c6217ae2fe None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:49.229307 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0d9871-3dd7-4862-aee9-59c6217ae2fe None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:49.247960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-2e0d9871-3dd7-4862-aee9-59c6217ae2fe None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 31 14:44:49.247960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:49.247960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:49.247960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:49.247960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:49.247960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:49.247960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:49.247960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:49.247960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 31 14:44:49.247960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:49.247960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:49.247960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:49.247960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:49.247960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:49.247960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:49.247960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:49.247960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:49.249105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:49.249105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:49.249105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:49.249105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:49.249105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:49.249105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:49.249105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:49.249105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:49.249105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:49.249105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:49.249105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:49.249105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:49.249105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:49.249105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:49.249105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:49.249105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:49.250159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:49.250159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:49.250159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:49.250159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:49.250159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:49.250159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:49.250159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:49.250159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:49.250159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:49.250159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:49.250159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:49.250159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:49.250159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 31 14:44:49.250159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:49.250159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 368/728] 10.222.0.36 () {56 vars in 1105 bytes} [Mon Oct 31 14:44:49 2022] GET /identity/v3/services/image => generated 86 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:49.255781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0d9871-3dd7-4862-aee9-59c6217ae2fe None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:49.285534 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 361/729] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:49 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 31 14:44:49.425226 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0d9871-3dd7-4862-aee9-59c6217ae2fe None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:49.472860 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d1c00e-7a63-4b08-a0c6-938b7deafd86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:49.473470 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d1c00e-7a63-4b08-a0c6-938b7deafd86 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:49.473748 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d1c00e-7a63-4b08-a0c6-938b7deafd86 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:49.474018 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d1c00e-7a63-4b08-a0c6-938b7deafd86 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:49.474648 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d1c00e-7a63-4b08-a0c6-938b7deafd86 None admin] RBAC: Adding query filter params (name=image) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:49.474948 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d1c00e-7a63-4b08-a0c6-938b7deafd86 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:49.476337 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d1c00e-7a63-4b08-a0c6-938b7deafd86 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:49.477400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:49.482744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 369/730] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:49 2022] GET /identity/v3/services?name=image => generated 122 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:49.489372 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad32e61-14ea-4c61-bf22-cc966bd45c1d None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:49.614644 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad32e61-14ea-4c61-bf22-cc966bd45c1d None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:49.620393 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d1c00e-7a63-4b08-a0c6-938b7deafd86 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:49.652340 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd38add-f5c6-45cf-99a2-7c66e2d61f8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:49.653065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd38add-f5c6-45cf-99a2-7c66e2d61f8a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:49.653232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd38add-f5c6-45cf-99a2-7c66e2d61f8a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:49.653498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd38add-f5c6-45cf-99a2-7c66e2d61f8a None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:49.654135 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd38add-f5c6-45cf-99a2-7c66e2d61f8a None admin] RBAC: Adding query filter params (type=image) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:49.654440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd38add-f5c6-45cf-99a2-7c66e2d61f8a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:49.655839 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd38add-f5c6-45cf-99a2-7c66e2d61f8a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:49.663596 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 362/731] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:49 2022] GET /identity/v3/services?type=image => generated 122 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:49.743504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d1c00e-7a63-4b08-a0c6-938b7deafd86 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:49.781835 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bd1a08-7271-4d1b-ad2d-7acf499cb643 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:49.782480 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bd1a08-7271-4d1b-ad2d-7acf499cb643 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:49.782757 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bd1a08-7271-4d1b-ad2d-7acf499cb643 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:49.783029 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bd1a08-7271-4d1b-ad2d-7acf499cb643 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:49.783651 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bd1a08-7271-4d1b-ad2d-7acf499cb643 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:49.785049 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bd1a08-7271-4d1b-ad2d-7acf499cb643 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:49.795429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-58bd1a08-7271-4d1b-ad2d-7acf499cb643 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 31 14:44:49.795429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:49.795429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:49.795429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:49.795429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:49.795429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:49.795429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:49.795429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:49.795429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 31 14:44:49.795429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:49.795429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:49.795429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:49.795429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:49.795429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:49.795429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:49.795429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:49.795429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:49.796572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:49.796572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:49.796572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:49.796572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:49.796572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:49.796572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:49.796572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:49.796572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:49.796572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:49.796572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:49.796572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:49.796572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:49.796572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:49.796572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:49.796572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:49.796572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:49.797409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:49.797409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:49.797409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:49.797409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:49.797409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:49.797409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:49.797409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:49.797409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:49.797409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:49.797409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:49.797409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:49.797409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:49.797409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 31 14:44:49.797409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:49.797409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 370/732] 10.222.0.36 () {56 vars in 1126 bytes} [Mon Oct 31 14:44:49 2022] GET /identity/v3/services/object-store => generated 93 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:49.803027 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd38add-f5c6-45cf-99a2-7c66e2d61f8a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:49.832383 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:49.918847 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd38add-f5c6-45cf-99a2-7c66e2d61f8a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:49.955663 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424a12c7-5d0c-4a67-9f53-cc6431fd2752 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:49.956272 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a12c7-5d0c-4a67-9f53-cc6431fd2752 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:49.956554 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a12c7-5d0c-4a67-9f53-cc6431fd2752 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:49.956830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a12c7-5d0c-4a67-9f53-cc6431fd2752 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:49.957466 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424a12c7-5d0c-4a67-9f53-cc6431fd2752 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:49.957772 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424a12c7-5d0c-4a67-9f53-cc6431fd2752 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:49.959145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424a12c7-5d0c-4a67-9f53-cc6431fd2752 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:49.965856 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 363/733] 10.222.0.36 () {56 vars in 1122 bytes} [Mon Oct 31 14:44:49 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:49.972620 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bd1a08-7271-4d1b-ad2d-7acf499cb643 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:50.039652 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:50.084015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bd1a08-7271-4d1b-ad2d-7acf499cb643 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:50.119414 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7502f926-5861-4f95-98d4-1da36b9c572e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:50.120381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7502f926-5861-4f95-98d4-1da36b9c572e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:50.120855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7502f926-5861-4f95-98d4-1da36b9c572e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:50.121347 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7502f926-5861-4f95-98d4-1da36b9c572e None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:50.122181 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7502f926-5861-4f95-98d4-1da36b9c572e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:50.122675 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7502f926-5861-4f95-98d4-1da36b9c572e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:50.124290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7502f926-5861-4f95-98d4-1da36b9c572e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:50.134300 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 371/734] 10.222.0.36 () {56 vars in 1122 bytes} [Mon Oct 31 14:44:49 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:50.136560 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424a12c7-5d0c-4a67-9f53-cc6431fd2752 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:50.232929 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:50.249415 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424a12c7-5d0c-4a67-9f53-cc6431fd2752 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:50.286388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634cb648-3759-4e9a-bed2-6871675d48da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:50.286996 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634cb648-3759-4e9a-bed2-6871675d48da None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:50.287277 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634cb648-3759-4e9a-bed2-6871675d48da None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:50.287574 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634cb648-3759-4e9a-bed2-6871675d48da None admin] PATH_INFO: `/v3/endpoints` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:50.288368 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634cb648-3759-4e9a-bed2-6871675d48da None admin] RBAC: Adding query filter params (interface=admin, service_id=d3ae3a35bd894e87b555d9135ffc42c6, region_id=RegionOne) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:50.288680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634cb648-3759-4e9a-bed2-6871675d48da None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:50.290108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634cb648-3759-4e9a-bed2-6871675d48da None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:50.296183 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 364/735] 10.222.0.36 () {56 vars in 1249 bytes} [Mon Oct 31 14:44:50 2022] GET /identity/v3/endpoints?service_id=d3ae3a35bd894e87b555d9135ffc42c6&interface=admin®ion_id=RegionOne => generated 193 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:50.301890 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7502f926-5861-4f95-98d4-1da36b9c572e None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:50.408674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7502f926-5861-4f95-98d4-1da36b9c572e None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:50.436292 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:50.446331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f1374f-9b3e-4df4-8b5e-9b9f56d65b31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:50.447231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f1374f-9b3e-4df4-8b5e-9b9f56d65b31 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:50.447872 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f1374f-9b3e-4df4-8b5e-9b9f56d65b31 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:50.448343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f1374f-9b3e-4df4-8b5e-9b9f56d65b31 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:50.449196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f1374f-9b3e-4df4-8b5e-9b9f56d65b31 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:50.450770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f1374f-9b3e-4df4-8b5e-9b9f56d65b31 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:50.463691 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 372/736] 10.222.0.36 () {56 vars in 1087 bytes} [Mon Oct 31 14:44:50 2022] GET /identity/v3/services => generated 1497 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:44:50.907399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23567560-8398-489b-8704-ce1c9002bd98 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:50.907953 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23567560-8398-489b-8704-ce1c9002bd98 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:50.907953 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23567560-8398-489b-8704-ce1c9002bd98 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:50.909567 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 365/737] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:50.914895 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d86707-cb4a-46c4-8e87-b19ff095dd1b None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:50.915178 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d86707-cb4a-46c4-8e87-b19ff095dd1b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:50.915452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d86707-cb4a-46c4-8e87-b19ff095dd1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:50.934929 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-68d86707-cb4a-46c4-8e87-b19ff095dd1b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:50.936151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-68d86707-cb4a-46c4-8e87-b19ff095dd1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:51.011589 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 373/738] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:50 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:51.282058 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23567560-8398-489b-8704-ce1c9002bd98 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:51.388287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23567560-8398-489b-8704-ce1c9002bd98 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:51.411082 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b3766c-650e-4a49-ad72-6daa6a89abfd None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:51.411644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b3766c-650e-4a49-ad72-6daa6a89abfd None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:51.411644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b3766c-650e-4a49-ad72-6daa6a89abfd None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:51.413425 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 374/739] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:44:51.419000 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde35dc2-e2f3-46a0-af1f-e22b1a29de82 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:51.419276 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde35dc2-e2f3-46a0-af1f-e22b1a29de82 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:51.419561 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde35dc2-e2f3-46a0-af1f-e22b1a29de82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:51.433005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9910bfa9-e91d-467a-b7bc-f9d5d2a52b8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:51.433628 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9910bfa9-e91d-467a-b7bc-f9d5d2a52b8e None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:51.433907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9910bfa9-e91d-467a-b7bc-f9d5d2a52b8e None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:51.434174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9910bfa9-e91d-467a-b7bc-f9d5d2a52b8e None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:51.434828 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9910bfa9-e91d-467a-b7bc-f9d5d2a52b8e None admin] RBAC: Authorizing `identity:create_service()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:51.436227 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9910bfa9-e91d-467a-b7bc-f9d5d2a52b8e None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:51.438734 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-bde35dc2-e2f3-46a0-af1f-e22b1a29de82 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:51.440214 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-bde35dc2-e2f3-46a0-af1f-e22b1a29de82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:51.454605 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 366/740] 10.222.0.36 () {60 vars in 1141 bytes} [Mon Oct 31 14:44:51 2022] POST /identity/v3/services => generated 242 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:51.552073 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 375/741] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:51 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:51.827701 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9910bfa9-e91d-467a-b7bc-f9d5d2a52b8e None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:51.843453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:51.890751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df446909-d165-4553-836e-6fe4fc416852 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:51.891393 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df446909-d165-4553-836e-6fe4fc416852 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:51.891985 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df446909-d165-4553-836e-6fe4fc416852 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:51.894005 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 376/742] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:44:51.898669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ac9003-c616-4f93-85a1-aff1a24bc807 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:51.898957 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ac9003-c616-4f93-85a1-aff1a24bc807 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:51.899232 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ac9003-c616-4f93-85a1-aff1a24bc807 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:51.922583 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-d6ac9003-c616-4f93-85a1-aff1a24bc807 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:51.923864 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d6ac9003-c616-4f93-85a1-aff1a24bc807 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:51.943303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9910bfa9-e91d-467a-b7bc-f9d5d2a52b8e None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:51.977697 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:51.979422 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f340b152-07a7-4bf5-8d51-1bbf71169d07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:51.980107 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f340b152-07a7-4bf5-8d51-1bbf71169d07 None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:51.980387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f340b152-07a7-4bf5-8d51-1bbf71169d07 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:51.980691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f340b152-07a7-4bf5-8d51-1bbf71169d07 None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:51.981335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f340b152-07a7-4bf5-8d51-1bbf71169d07 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:51.983303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f340b152-07a7-4bf5-8d51-1bbf71169d07 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:51.985982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: WARNING keystone.server.flask.common [None req-f340b152-07a7-4bf5-8d51-1bbf71169d07 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 31 14:44:52.008752 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 377/743] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:51 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:52.008979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.sql.core [None req-f340b152-07a7-4bf5-8d51-1bbf71169d07 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 31 14:44:52.008979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 31 14:44:52.008979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [parameters: {'id': '46069c51a82d47fcaec378f3e3f84835', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 31 14:44:52.008979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=96564) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 31 14:44:52.017505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-f340b152-07a7-4bf5-8d51-1bbf71169d07 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 46069c51a82d47fcaec378f3e3f84835.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 46069c51a82d47fcaec378f3e3f84835. Oct 31 14:44:52.017505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:52.017505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 31 14:44:52.017505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 31 14:44:52.017505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 31 14:44:52.017505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 31 14:44:52.017505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 31 14:44:52.017505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application result = self._query(query) Oct 31 14:44:52.017505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 31 14:44:52.017505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application conn.query(q) Oct 31 14:44:52.017505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 31 14:44:52.017505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 31 14:44:52.017505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application result.read() Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 31 14:44:52.018392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 31 14:44:52.019378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 31 14:44:52.019378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application next(self.gen) Oct 31 14:44:52.019378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 31 14:44:52.019378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application yield resource Oct 31 14:44:52.019378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 31 14:44:52.019378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application next(self.gen) Oct 31 14:44:52.019378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 31 14:44:52.019378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.session.rollback() Oct 31 14:44:52.019378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 31 14:44:52.019378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.force_reraise() Oct 31 14:44:52.019378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 31 14:44:52.019378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise self.value Oct 31 14:44:52.019378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 31 14:44:52.019378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 31 14:44:52.019378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 31 14:44:52.019378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application session.commit() Oct 31 14:44:52.019378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 31 14:44:52.020339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 31 14:44:52.020339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 31 14:44:52.020339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._prepare_impl() Oct 31 14:44:52.020339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 31 14:44:52.020339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.session.flush() Oct 31 14:44:52.020339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 31 14:44:52.020339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._flush(objects) Oct 31 14:44:52.020339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 31 14:44:52.020339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 31 14:44:52.020339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 31 14:44:52.020339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application compat.raise_( Oct 31 14:44:52.020339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 31 14:44:52.020339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception Oct 31 14:44:52.020339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 31 14:44:52.020339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application flush_context.execute() Oct 31 14:44:52.020339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 31 14:44:52.020339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rec.execute(self) Oct 31 14:44:52.021270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 31 14:44:52.021270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 31 14:44:52.021270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 31 14:44:52.021270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 31 14:44:52.021270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 31 14:44:52.021270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 31 14:44:52.021270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 31 14:44:52.021270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 31 14:44:52.021270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 31 14:44:52.021270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 31 14:44:52.021270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 31 14:44:52.021270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 31 14:44:52.021270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 31 14:44:52.021270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 31 14:44:52.021270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 31 14:44:52.022079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 31 14:44:52.022079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 31 14:44:52.022079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception Oct 31 14:44:52.022079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 31 14:44:52.022079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 31 14:44:52.022079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 31 14:44:52.022079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 31 14:44:52.022079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 31 14:44:52.022079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application result = self._query(query) Oct 31 14:44:52.022079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 31 14:44:52.022079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application conn.query(q) Oct 31 14:44:52.022079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 31 14:44:52.022079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 31 14:44:52.022079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 31 14:44:52.022079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application result.read() Oct 31 14:44:52.022079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 31 14:44:52.022079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 31 14:44:52.022971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 31 14:44:52.022971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 31 14:44:52.022971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 31 14:44:52.022971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 31 14:44:52.022971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 31 14:44:52.022971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 31 14:44:52.022971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 31 14:44:52.022971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 31 14:44:52.022971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [parameters: {'id': '46069c51a82d47fcaec378f3e3f84835', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 31 14:44:52.022971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 31 14:44:52.022971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:52.022971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:52.022971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:52.022971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:52.022971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 31 14:44:52.027948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 31 14:44:52.027948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 31 14:44:52.027948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 31 14:44:52.027948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 31 14:44:52.027948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:52.027948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:52.027948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:52.027948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:52.027948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:52.027948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:52.027948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:52.027948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:52.027948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:52.027948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:52.027948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:52.027948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:52.027948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:52.028875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:52.028875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 31 14:44:52.028875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 31 14:44:52.028875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:52.028875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:52.028875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 31 14:44:52.028875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 31 14:44:52.028875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 46069c51a82d47fcaec378f3e3f84835. Oct 31 14:44:52.028875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:52.028875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 367/744] 10.222.0.36 () {60 vars in 1140 bytes} [Mon Oct 31 14:44:51 2022] POST /identity/v3/projects => generated 262 bytes in 191 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:52.029420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ac9003-c616-4f93-85a1-aff1a24bc807 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:52.139485 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ac9003-c616-4f93-85a1-aff1a24bc807 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:52.173836 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3bac78-09d9-435e-a939-40721b27baf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:52.174468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3bac78-09d9-435e-a939-40721b27baf9 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:52.174758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3bac78-09d9-435e-a939-40721b27baf9 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:52.175021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3bac78-09d9-435e-a939-40721b27baf9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:52.183118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3bac78-09d9-435e-a939-40721b27baf9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:52.184603 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3bac78-09d9-435e-a939-40721b27baf9 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:52.196108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-0a3bac78-09d9-435e-a939-40721b27baf9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:52.196108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:52.196108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:52.196108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:52.196108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:52.196108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:52.196108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:52.196108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:52.196108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:52.196108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:52.196108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:52.196108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:52.196108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:44:52.196108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:44:52.196108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:44:52.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:44:52.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:52.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:52.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:52.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:52.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:52.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:52.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:52.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:52.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:52.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:52.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:52.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:52.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:52.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:52.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:52.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:52.198728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:44:52.198728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:44:52.198728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:44:52.198728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:44:52.198728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:44:52.198728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:44:52.198728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:44:52.198728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:52.198728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 378/745] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:52.205264 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f340b152-07a7-4bf5-8d51-1bbf71169d07 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:52.276212 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:52.317876 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f340b152-07a7-4bf5-8d51-1bbf71169d07 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:52.349121 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3bac78-09d9-435e-a939-40721b27baf9 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:52.364055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772c3fd8-6ec4-4ed3-801e-14f4e1190cd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:52.365373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772c3fd8-6ec4-4ed3-801e-14f4e1190cd8 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:52.365872 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772c3fd8-6ec4-4ed3-801e-14f4e1190cd8 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:52.366862 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772c3fd8-6ec4-4ed3-801e-14f4e1190cd8 None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:52.368095 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772c3fd8-6ec4-4ed3-801e-14f4e1190cd8 None admin] RBAC: Adding query filter params (name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:52.368879 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772c3fd8-6ec4-4ed3-801e-14f4e1190cd8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:52.371725 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772c3fd8-6ec4-4ed3-801e-14f4e1190cd8 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:52.388024 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:52.407726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 368/746] 10.222.0.36 () {56 vars in 1112 bytes} [Mon Oct 31 14:44:52 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:52.476284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3bac78-09d9-435e-a939-40721b27baf9 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:52.515457 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7adee55-0b21-4d05-b52c-6aa02dad6e32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:52.516101 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7adee55-0b21-4d05-b52c-6aa02dad6e32 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:52.516381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7adee55-0b21-4d05-b52c-6aa02dad6e32 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:52.516651 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7adee55-0b21-4d05-b52c-6aa02dad6e32 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:52.517324 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7adee55-0b21-4d05-b52c-6aa02dad6e32 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:52.518694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7adee55-0b21-4d05-b52c-6aa02dad6e32 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:52.547733 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 379/747] 10.222.0.36 () {60 vars in 1141 bytes} [Mon Oct 31 14:44:52 2022] POST /identity/v3/services => generated 242 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:52.745986 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e1c517-b953-4107-a42c-e270426f72c5 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:52.746318 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e1c517-b953-4107-a42c-e270426f72c5 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:52.746492 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e1c517-b953-4107-a42c-e270426f72c5 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:52.747966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 369/748] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:52.753671 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb22b5e6-bf5f-42d9-86cb-253580120a24 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:52.754208 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb22b5e6-bf5f-42d9-86cb-253580120a24 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:52.754747 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb22b5e6-bf5f-42d9-86cb-253580120a24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:52.775928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-eb22b5e6-bf5f-42d9-86cb-253580120a24 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:52.777491 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-eb22b5e6-bf5f-42d9-86cb-253580120a24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:52.855407 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 380/749] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:52 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:53.132785 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e1c517-b953-4107-a42c-e270426f72c5 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:53.247379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e1c517-b953-4107-a42c-e270426f72c5 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:53.276621 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:53.286790 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b13e309-315f-41ea-88ea-a9e359c771c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:53.287703 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b13e309-315f-41ea-88ea-a9e359c771c0 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:53.288146 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b13e309-315f-41ea-88ea-a9e359c771c0 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:53.288564 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b13e309-315f-41ea-88ea-a9e359c771c0 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:53.289301 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b13e309-315f-41ea-88ea-a9e359c771c0 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:53.290899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b13e309-315f-41ea-88ea-a9e359c771c0 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:53.300535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-5b13e309-315f-41ea-88ea-a9e359c771c0 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 31 14:44:53.300535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:53.300535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:53.300535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:53.300535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:53.300535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:53.300535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:53.300535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:53.300535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 31 14:44:53.300535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:53.300535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:53.300535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:53.300535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:53.300535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:53.300535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:53.300535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:53.300535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:53.301762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:53.301762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:53.301762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:53.301762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:53.301762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:53.301762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:53.301762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:53.301762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:53.301762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:53.301762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:53.301762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:53.301762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:53.301762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:53.301762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:53.301762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:53.301762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:53.302882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:53.302882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:53.302882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:53.302882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:53.302882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:53.302882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:53.302882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:53.302882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:53.302882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:53.302882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:53.302882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:53.302882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:53.302882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 31 14:44:53.302882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:53.305439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 370/750] 10.222.0.36 () {56 vars in 1126 bytes} [Mon Oct 31 14:44:53 2022] GET /identity/v3/services/object-store => generated 93 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:53.312975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb22b5e6-bf5f-42d9-86cb-253580120a24 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:53.340150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:53.425070 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb22b5e6-bf5f-42d9-86cb-253580120a24 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:53.463473 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80811997-44eb-4fd0-a38d-e87b75df398d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:53.464354 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80811997-44eb-4fd0-a38d-e87b75df398d None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:53.464806 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80811997-44eb-4fd0-a38d-e87b75df398d None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:53.465238 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80811997-44eb-4fd0-a38d-e87b75df398d None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:53.466010 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80811997-44eb-4fd0-a38d-e87b75df398d None admin] RBAC: Adding query filter params (name=object-store) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:53.466456 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80811997-44eb-4fd0-a38d-e87b75df398d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:53.468240 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80811997-44eb-4fd0-a38d-e87b75df398d None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:53.473769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 381/751] 10.222.0.36 () {56 vars in 1122 bytes} [Mon Oct 31 14:44:53 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:53.480780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b13e309-315f-41ea-88ea-a9e359c771c0 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:53.587689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b13e309-315f-41ea-88ea-a9e359c771c0 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:53.623309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ceba086-bc79-467c-80a2-d0f6d312096c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:53.624300 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ceba086-bc79-467c-80a2-d0f6d312096c None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:53.624800 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ceba086-bc79-467c-80a2-d0f6d312096c None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:53.625269 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ceba086-bc79-467c-80a2-d0f6d312096c None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:53.626085 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ceba086-bc79-467c-80a2-d0f6d312096c None admin] RBAC: Adding query filter params (type=object-store) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:53.626579 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ceba086-bc79-467c-80a2-d0f6d312096c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:53.628159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ceba086-bc79-467c-80a2-d0f6d312096c None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:53.633725 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 371/752] 10.222.0.36 () {56 vars in 1122 bytes} [Mon Oct 31 14:44:53 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:53.640047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80811997-44eb-4fd0-a38d-e87b75df398d None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:53.735678 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:53.754035 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80811997-44eb-4fd0-a38d-e87b75df398d None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:53.782636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f277ca1-276a-49e5-8a57-bda97c7535bc None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:53.783234 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f277ca1-276a-49e5-8a57-bda97c7535bc None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:53.783774 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f277ca1-276a-49e5-8a57-bda97c7535bc None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:53.785323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 372/753] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:44:53.793281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ff31f0-fb36-4eee-9ecf-d36fbda71214 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:53.793570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ff31f0-fb36-4eee-9ecf-d36fbda71214 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:53.793847 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ff31f0-fb36-4eee-9ecf-d36fbda71214 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:53.794552 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d610fb-9fa2-4b61-ba81-c244c9cf430b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:53.795396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d610fb-9fa2-4b61-ba81-c244c9cf430b None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:53.795920 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d610fb-9fa2-4b61-ba81-c244c9cf430b None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:53.796401 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d610fb-9fa2-4b61-ba81-c244c9cf430b None admin] PATH_INFO: `/v3/endpoints` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:53.797266 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d610fb-9fa2-4b61-ba81-c244c9cf430b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:53.798820 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d610fb-9fa2-4b61-ba81-c244c9cf430b None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:53.818229 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-25ff31f0-fb36-4eee-9ecf-d36fbda71214 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:53.819877 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-25ff31f0-fb36-4eee-9ecf-d36fbda71214 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:53.820931 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 382/754] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:44:53 2022] POST /identity/v3/endpoints => generated 325 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:53.889056 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:53.918236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 373/755] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:53 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 31 14:44:54.202004 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d610fb-9fa2-4b61-ba81-c244c9cf430b None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:54.309579 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d610fb-9fa2-4b61-ba81-c244c9cf430b None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:54.345559 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37acb05-1b32-48ec-bb1a-45eec605bf43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:54.346195 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37acb05-1b32-48ec-bb1a-45eec605bf43 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:54.346475 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37acb05-1b32-48ec-bb1a-45eec605bf43 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:54.346751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37acb05-1b32-48ec-bb1a-45eec605bf43 None admin] PATH_INFO: `/v3/services/compute` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:54.347327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37acb05-1b32-48ec-bb1a-45eec605bf43 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:54.348788 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37acb05-1b32-48ec-bb1a-45eec605bf43 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:54.358318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-b37acb05-1b32-48ec-bb1a-45eec605bf43 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 31 14:44:54.358318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:54.358318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:54.358318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:54.358318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:54.358318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:54.358318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:54.358318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:54.358318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 31 14:44:54.358318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:54.358318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:54.358318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:54.358318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:54.358318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:54.358318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:54.358318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:54.358318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:54.359309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:54.359309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:54.359309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:54.359309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:54.359309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:54.359309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:54.359309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:54.359309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:54.359309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:54.359309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:54.359309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:54.359309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:54.359309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:54.359309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:54.359309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:54.359309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:54.360567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:54.360567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:54.360567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:54.360567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:54.360567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:54.360567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:54.360567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:54.360567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:54.360567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:54.360567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:54.360567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:54.360567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:54.360567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 31 14:44:54.360567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:54.360567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 383/756] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:54 2022] GET /identity/v3/services/compute => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:54.365799 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ff31f0-fb36-4eee-9ecf-d36fbda71214 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:54.484009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ff31f0-fb36-4eee-9ecf-d36fbda71214 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:54.522860 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605fa407-fe01-405d-87cb-52a995c3a486 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:54.523476 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605fa407-fe01-405d-87cb-52a995c3a486 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:54.523788 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605fa407-fe01-405d-87cb-52a995c3a486 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:54.524064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605fa407-fe01-405d-87cb-52a995c3a486 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:54.524702 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605fa407-fe01-405d-87cb-52a995c3a486 None admin] RBAC: Adding query filter params (name=compute) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:54.525008 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605fa407-fe01-405d-87cb-52a995c3a486 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:54.531571 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605fa407-fe01-405d-87cb-52a995c3a486 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:54.533457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 374/757] 10.222.0.36 () {56 vars in 1112 bytes} [Mon Oct 31 14:44:54 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:54.541635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37acb05-1b32-48ec-bb1a-45eec605bf43 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:54.555926 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:54.649955 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37acb05-1b32-48ec-bb1a-45eec605bf43 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:54.687006 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206a5cd1-e3ed-4460-b89c-72808b2acb37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:54.687649 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206a5cd1-e3ed-4460-b89c-72808b2acb37 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:54.687931 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206a5cd1-e3ed-4460-b89c-72808b2acb37 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:54.688207 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206a5cd1-e3ed-4460-b89c-72808b2acb37 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:54.688899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206a5cd1-e3ed-4460-b89c-72808b2acb37 None admin] RBAC: Adding query filter params (type=compute) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:54.689207 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206a5cd1-e3ed-4460-b89c-72808b2acb37 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:54.690621 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206a5cd1-e3ed-4460-b89c-72808b2acb37 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:54.696292 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 384/758] 10.222.0.36 () {56 vars in 1112 bytes} [Mon Oct 31 14:44:54 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:54.703752 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605fa407-fe01-405d-87cb-52a995c3a486 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:54.762173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:54.827063 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8774e41-8885-43db-91c9-7f72beff75e1 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:54.827388 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8774e41-8885-43db-91c9-7f72beff75e1 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:54.827646 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8774e41-8885-43db-91c9-7f72beff75e1 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:54.828140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605fa407-fe01-405d-87cb-52a995c3a486 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:54.829224 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 385/759] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:54.834132 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacb87a4-18cc-4468-9b7b-f422eedfc2d9 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:54.834416 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacb87a4-18cc-4468-9b7b-f422eedfc2d9 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:54.834696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacb87a4-18cc-4468-9b7b-f422eedfc2d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:54.852967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-eacb87a4-18cc-4468-9b7b-f422eedfc2d9 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:54.854204 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-eacb87a4-18cc-4468-9b7b-f422eedfc2d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:54.866395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7848aa51-d92d-4ca5-bf80-c1df06d9ca8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:54.867009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7848aa51-d92d-4ca5-bf80-c1df06d9ca8b None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:54.867297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7848aa51-d92d-4ca5-bf80-c1df06d9ca8b None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:54.867711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7848aa51-d92d-4ca5-bf80-c1df06d9ca8b None admin] PATH_INFO: `/v3/endpoints` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:54.868535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7848aa51-d92d-4ca5-bf80-c1df06d9ca8b None admin] RBAC: Adding query filter params (interface=public, service_id=207b64759eff4518beb32c79a455dcb3, region_id=RegionOne) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:54.869397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7848aa51-d92d-4ca5-bf80-c1df06d9ca8b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:54.871385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7848aa51-d92d-4ca5-bf80-c1df06d9ca8b None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:54.877501 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 375/760] 10.222.0.36 () {56 vars in 1251 bytes} [Mon Oct 31 14:44:54 2022] GET /identity/v3/endpoints?service_id=207b64759eff4518beb32c79a455dcb3&interface=public®ion_id=RegionOne => generated 194 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:54.883241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7848aa51-d92d-4ca5-bf80-c1df06d9ca8b None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:54.923683 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 386/761] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:54 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 31 14:44:54.925280 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be32025d-c080-43fb-8b00-b930804c2c5a None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:54.925720 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be32025d-c080-43fb-8b00-b930804c2c5a None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:54.925995 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be32025d-c080-43fb-8b00-b930804c2c5a None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:54.927373 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 387/762] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:54.932989 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4060d1cd-26f0-42ca-9933-9102f2842197 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:54.933270 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4060d1cd-26f0-42ca-9933-9102f2842197 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:54.933547 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4060d1cd-26f0-42ca-9933-9102f2842197 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:54.940582 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:54.954933 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-4060d1cd-26f0-42ca-9933-9102f2842197 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:54.956281 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-4060d1cd-26f0-42ca-9933-9102f2842197 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:55.001130 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7848aa51-d92d-4ca5-bf80-c1df06d9ca8b None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:55.011145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:55.034582 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 388/763] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:54 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 31 14:44:55.042614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d5a3e1-071b-4a4d-9977-b4e08813e1bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:55.043470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d5a3e1-071b-4a4d-9977-b4e08813e1bd None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:55.043988 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d5a3e1-071b-4a4d-9977-b4e08813e1bd None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:55.044460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d5a3e1-071b-4a4d-9977-b4e08813e1bd None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:55.045287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d5a3e1-071b-4a4d-9977-b4e08813e1bd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:55.046930 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d5a3e1-071b-4a4d-9977-b4e08813e1bd None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:55.054242 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 376/764] 10.222.0.36 () {56 vars in 1087 bytes} [Mon Oct 31 14:44:54 2022] GET /identity/v3/services => generated 1957 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:44:55.206961 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4060d1cd-26f0-42ca-9933-9102f2842197 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:55.310756 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d5a3e1-071b-4a4d-9977-b4e08813e1bd None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:55.313530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4060d1cd-26f0-42ca-9933-9102f2842197 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:55.354923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ccdda0-3bd4-4be9-90bf-4ef9a00cc524 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:55.355571 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ccdda0-3bd4-4be9-90bf-4ef9a00cc524 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:55.355862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ccdda0-3bd4-4be9-90bf-4ef9a00cc524 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:55.356134 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ccdda0-3bd4-4be9-90bf-4ef9a00cc524 None admin] PATH_INFO: `/v3/services/image` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:55.356712 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ccdda0-3bd4-4be9-90bf-4ef9a00cc524 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:55.358076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ccdda0-3bd4-4be9-90bf-4ef9a00cc524 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:55.360046 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:55.367700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-c0ccdda0-3bd4-4be9-90bf-4ef9a00cc524 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 31 14:44:55.367700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:55.367700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:55.367700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:55.367700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:55.367700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:55.367700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:55.367700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:55.367700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 31 14:44:55.367700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:55.367700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:55.367700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:55.367700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:55.367700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:55.367700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:55.367700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:55.367700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:55.369055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:55.369055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:55.369055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:55.369055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:55.369055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:55.369055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:55.369055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:55.369055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:55.369055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:55.369055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:55.369055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:55.369055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:55.369055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:55.369055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:55.369055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:55.369055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:55.370313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:55.370313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:55.370313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:55.370313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:55.370313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:55.370313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:55.370313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:55.370313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:55.370313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:55.370313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:55.370313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:55.370313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:55.370313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 31 14:44:55.370313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:44:55.371270 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 389/765] 10.222.0.36 () {56 vars in 1105 bytes} [Mon Oct 31 14:44:55 2022] GET /identity/v3/services/image => generated 86 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:55.378532 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ccdda0-3bd4-4be9-90bf-4ef9a00cc524 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:55.432022 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d5a3e1-071b-4a4d-9977-b4e08813e1bd None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:55.468570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de1a9c5-1a4e-4ff9-9283-a2e57fd404f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:55.469463 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1a9c5-1a4e-4ff9-9283-a2e57fd404f2 None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:55.469576 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1a9c5-1a4e-4ff9-9283-a2e57fd404f2 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:55.469778 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1a9c5-1a4e-4ff9-9283-a2e57fd404f2 None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:55.470479 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de1a9c5-1a4e-4ff9-9283-a2e57fd404f2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:55.471955 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de1a9c5-1a4e-4ff9-9283-a2e57fd404f2 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:55.487853 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.sql.core [None req-5de1a9c5-1a4e-4ff9-9283-a2e57fd404f2 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 31 14:44:55.487853 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 31 14:44:55.487853 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [parameters: {'user_id': '6a5af8cc186c4774be272a7b0abcd553', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 31 14:44:55.487853 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=96564) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 31 14:44:55.489848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ccdda0-3bd4-4be9-90bf-4ef9a00cc524 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:55.496807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-5de1a9c5-1a4e-4ff9-9283-a2e57fd404f2 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 31 14:44:55.496807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:55.496807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 31 14:44:55.496807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 31 14:44:55.496807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 31 14:44:55.496807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 31 14:44:55.496807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 31 14:44:55.496807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application result = self._query(query) Oct 31 14:44:55.496807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 31 14:44:55.496807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application conn.query(q) Oct 31 14:44:55.496807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 31 14:44:55.496807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 31 14:44:55.496807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 31 14:44:55.496807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application result.read() Oct 31 14:44:55.496807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 31 14:44:55.497696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application next(self.gen) Oct 31 14:44:55.498899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 31 14:44:55.498899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application yield resource Oct 31 14:44:55.498899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 31 14:44:55.498899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application next(self.gen) Oct 31 14:44:55.498899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 31 14:44:55.498899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.session.rollback() Oct 31 14:44:55.498899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 31 14:44:55.498899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.force_reraise() Oct 31 14:44:55.498899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 31 14:44:55.498899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise self.value Oct 31 14:44:55.498899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 31 14:44:55.498899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 31 14:44:55.498899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 31 14:44:55.498899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application session.commit() Oct 31 14:44:55.498899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 31 14:44:55.498899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 31 14:44:55.498899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 31 14:44:55.500079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._prepare_impl() Oct 31 14:44:55.500079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 31 14:44:55.500079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.session.flush() Oct 31 14:44:55.500079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 31 14:44:55.500079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._flush(objects) Oct 31 14:44:55.500079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 31 14:44:55.500079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 31 14:44:55.500079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 31 14:44:55.500079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application compat.raise_( Oct 31 14:44:55.500079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 31 14:44:55.500079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception Oct 31 14:44:55.500079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 31 14:44:55.500079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application flush_context.execute() Oct 31 14:44:55.500079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 31 14:44:55.500079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rec.execute(self) Oct 31 14:44:55.500079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 31 14:44:55.500079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 31 14:44:55.501233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 31 14:44:55.501233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 31 14:44:55.501233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 31 14:44:55.501233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 31 14:44:55.501233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 31 14:44:55.501233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 31 14:44:55.501233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 31 14:44:55.501233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 31 14:44:55.501233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 31 14:44:55.501233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 31 14:44:55.501233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 31 14:44:55.501233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 31 14:44:55.501233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 31 14:44:55.501233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 31 14:44:55.501233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 31 14:44:55.501233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception Oct 31 14:44:55.502241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 31 14:44:55.502241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 31 14:44:55.502241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 31 14:44:55.502241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 31 14:44:55.502241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 31 14:44:55.502241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application result = self._query(query) Oct 31 14:44:55.502241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 31 14:44:55.502241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application conn.query(q) Oct 31 14:44:55.502241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 31 14:44:55.502241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 31 14:44:55.502241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 31 14:44:55.502241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application result.read() Oct 31 14:44:55.502241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 31 14:44:55.502241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 31 14:44:55.502241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 31 14:44:55.502241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 31 14:44:55.503262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 31 14:44:55.503262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 31 14:44:55.503262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 31 14:44:55.503262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 31 14:44:55.503262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 31 14:44:55.503262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 31 14:44:55.503262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [parameters: {'user_id': '6a5af8cc186c4774be272a7b0abcd553', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 31 14:44:55.503262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 31 14:44:55.503262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:55.503262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:55.503262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:55.503262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:55.503262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:55.503262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:55.503262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:55.504246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:55.504246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:55.504246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:55.504246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:55.504246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:55.504246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:55.504246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:55.504246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 31 14:44:55.504246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 31 14:44:55.504246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:55.504246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:55.504246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:55.504246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:55.504246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:55.504246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:55.504246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 31 14:44:55.504246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 31 14:44:55.505850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 31 14:44:55.505850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 31 14:44:55.505850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 31 14:44:55.505850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 31 14:44:55.505850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 31 14:44:55.505850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:55.505850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 377/766] 10.222.0.36 () {60 vars in 1131 bytes} [Mon Oct 31 14:44:55 2022] POST /identity/v3/users => generated 160 bytes in 188 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:44:55.506598 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de1a9c5-1a4e-4ff9-9283-a2e57fd404f2 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:55.547611 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:55.550881 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91d424d-94bd-42a4-b70c-a4762f10d248 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:55.551958 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91d424d-94bd-42a4-b70c-a4762f10d248 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:55.552468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91d424d-94bd-42a4-b70c-a4762f10d248 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:55.552942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91d424d-94bd-42a4-b70c-a4762f10d248 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:55.553840 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91d424d-94bd-42a4-b70c-a4762f10d248 None admin] RBAC: Adding query filter params (name=image) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:55.554315 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91d424d-94bd-42a4-b70c-a4762f10d248 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:55.557461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91d424d-94bd-42a4-b70c-a4762f10d248 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:55.569222 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 390/767] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:55 2022] GET /identity/v3/services?name=image => generated 122 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:55.570694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91d424d-94bd-42a4-b70c-a4762f10d248 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:55.625077 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de1a9c5-1a4e-4ff9-9283-a2e57fd404f2 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:55.638202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:55.665390 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02aad449-0767-46d1-89e4-e7c22c9aeaaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:55.666145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02aad449-0767-46d1-89e4-e7c22c9aeaaa None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:55.666496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02aad449-0767-46d1-89e4-e7c22c9aeaaa None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:55.666837 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02aad449-0767-46d1-89e4-e7c22c9aeaaa None admin] PATH_INFO: `/v3/users/cinder` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:55.675056 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02aad449-0767-46d1-89e4-e7c22c9aeaaa None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:55.676905 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02aad449-0767-46d1-89e4-e7c22c9aeaaa None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:55.682091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91d424d-94bd-42a4-b70c-a4762f10d248 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:55.690202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-02aad449-0767-46d1-89e4-e7c22c9aeaaa None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 31 14:44:55.690202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:55.690202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:55.690202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:55.690202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:55.690202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:55.690202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:55.690202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:55.690202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:55.690202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:55.690202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:55.690202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:55.690202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:44:55.690202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:44:55.690202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:44:55.690202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:55.691143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:55.692485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:55.692485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:55.692485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:44:55.692485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:44:55.692485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:44:55.692485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:44:55.692485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:44:55.692485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:44:55.692485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 31 14:44:55.692485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:55.692485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 378/768] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:44:55 2022] GET /identity/v3/users/cinder => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:55.697064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02aad449-0767-46d1-89e4-e7c22c9aeaaa None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:55.722578 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894defda-8776-4369-a80b-7cf147447679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:55.723624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894defda-8776-4369-a80b-7cf147447679 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:55.724128 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894defda-8776-4369-a80b-7cf147447679 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:55.724596 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894defda-8776-4369-a80b-7cf147447679 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:55.725437 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894defda-8776-4369-a80b-7cf147447679 None admin] RBAC: Adding query filter params (type=image) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:55.725963 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894defda-8776-4369-a80b-7cf147447679 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:55.727634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894defda-8776-4369-a80b-7cf147447679 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:55.733417 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 391/769] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:55 2022] GET /identity/v3/services?type=image => generated 350 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:55.739569 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894defda-8776-4369-a80b-7cf147447679 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:55.814941 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:55.824197 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02aad449-0767-46d1-89e4-e7c22c9aeaaa None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:55.863145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb04a63b-95e6-41dd-bc27-9fd67db61665 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:55.863803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb04a63b-95e6-41dd-bc27-9fd67db61665 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:55.864055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb04a63b-95e6-41dd-bc27-9fd67db61665 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:55.864326 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb04a63b-95e6-41dd-bc27-9fd67db61665 None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:55.865029 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb04a63b-95e6-41dd-bc27-9fd67db61665 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:55.865328 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb04a63b-95e6-41dd-bc27-9fd67db61665 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:55.866778 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb04a63b-95e6-41dd-bc27-9fd67db61665 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:55.871632 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894defda-8776-4369-a80b-7cf147447679 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:55.900712 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:55.910400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26022b5-a9c1-43e6-a063-bb0bf8fdca99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:55.911399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26022b5-a9c1-43e6-a063-bb0bf8fdca99 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:55.911981 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 379/770] 10.222.0.36 () {56 vars in 1101 bytes} [Mon Oct 31 14:44:55 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:55.912496 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26022b5-a9c1-43e6-a063-bb0bf8fdca99 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:55.912952 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26022b5-a9c1-43e6-a063-bb0bf8fdca99 None admin] PATH_INFO: `/v3/endpoints` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:55.913865 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26022b5-a9c1-43e6-a063-bb0bf8fdca99 None admin] RBAC: Adding query filter params (interface=public, service_id=89a1ad673d4547f38d09f18c45c41a51, region_id=RegionOne) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:55.914398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26022b5-a9c1-43e6-a063-bb0bf8fdca99 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:55.916112 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26022b5-a9c1-43e6-a063-bb0bf8fdca99 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:55.927545 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 392/771] 10.222.0.36 () {56 vars in 1251 bytes} [Mon Oct 31 14:44:55 2022] GET /identity/v3/endpoints?service_id=89a1ad673d4547f38d09f18c45c41a51&interface=public®ion_id=RegionOne => generated 194 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:55.929308 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb04a63b-95e6-41dd-bc27-9fd67db61665 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:56.059454 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb04a63b-95e6-41dd-bc27-9fd67db61665 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:56.100578 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166eca90-7a91-4662-9035-c1d6eb0b18ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:56.100578 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166eca90-7a91-4662-9035-c1d6eb0b18ed None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:56.100578 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166eca90-7a91-4662-9035-c1d6eb0b18ed None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:56.100901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166eca90-7a91-4662-9035-c1d6eb0b18ed None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:56.101543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166eca90-7a91-4662-9035-c1d6eb0b18ed None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:56.102937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166eca90-7a91-4662-9035-c1d6eb0b18ed None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:56.104029 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9db5a59-5651-4d3e-a944-689a966b7a99 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:56.104313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9db5a59-5651-4d3e-a944-689a966b7a99 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:56.104583 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9db5a59-5651-4d3e-a944-689a966b7a99 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:56.106040 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 393/772] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:56.111411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 380/773] 10.222.0.36 () {56 vars in 1087 bytes} [Mon Oct 31 14:44:55 2022] GET /identity/v3/services => generated 1957 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:44:56.111582 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc22c3b1-8aa0-442c-8116-a64a0957d387 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:56.111765 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc22c3b1-8aa0-442c-8116-a64a0957d387 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:56.112039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc22c3b1-8aa0-442c-8116-a64a0957d387 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:56.132850 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-dc22c3b1-8aa0-442c-8116-a64a0957d387 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:56.134095 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-dc22c3b1-8aa0-442c-8116-a64a0957d387 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:56.214576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 394/774] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:56 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:56.483050 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166eca90-7a91-4662-9035-c1d6eb0b18ed None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:56.503944 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:56.710323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166eca90-7a91-4662-9035-c1d6eb0b18ed None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:56.744665 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0824c383-de0f-456e-90b0-3450d6d9a43c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:56.745210 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0824c383-de0f-456e-90b0-3450d6d9a43c None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:56.745484 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0824c383-de0f-456e-90b0-3450d6d9a43c None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:56.745750 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0824c383-de0f-456e-90b0-3450d6d9a43c None admin] PATH_INFO: `/v3/domains/default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:56.746847 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0824c383-de0f-456e-90b0-3450d6d9a43c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:56.748317 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0824c383-de0f-456e-90b0-3450d6d9a43c None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:56.750032 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 381/775] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:56.755746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc22c3b1-8aa0-442c-8116-a64a0957d387 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:56.858612 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc22c3b1-8aa0-442c-8116-a64a0957d387 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:56.879747 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:56.895164 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9c6a3c-b344-433a-918e-14f5e4417578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:56.895815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9c6a3c-b344-433a-918e-14f5e4417578 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:56.896098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9c6a3c-b344-433a-918e-14f5e4417578 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:56.896373 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9c6a3c-b344-433a-918e-14f5e4417578 None admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:56.897036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9c6a3c-b344-433a-918e-14f5e4417578 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:56.898457 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9c6a3c-b344-433a-918e-14f5e4417578 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:56.931903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 395/776] 10.222.0.36 () {60 vars in 1141 bytes} [Mon Oct 31 14:44:56 2022] POST /identity/v3/projects => generated 311 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:57.304783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fd1607-91af-47e8-be85-91ec5c21edcd None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:57.305168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fd1607-91af-47e8-be85-91ec5c21edcd None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:57.305364 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fd1607-91af-47e8-be85-91ec5c21edcd None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:57.306926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 382/777] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:57.312002 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35df6d37-6528-4bd8-ac21-70555ca332bc None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:57.312290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35df6d37-6528-4bd8-ac21-70555ca332bc None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:57.312561 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35df6d37-6528-4bd8-ac21-70555ca332bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:57.330705 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-35df6d37-6528-4bd8-ac21-70555ca332bc None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:57.331907 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-35df6d37-6528-4bd8-ac21-70555ca332bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:57.402229 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 396/778] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:57 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:57.673148 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fd1607-91af-47e8-be85-91ec5c21edcd None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:57.761852 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:57.780523 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fd1607-91af-47e8-be85-91ec5c21edcd None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:57.816018 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f55019d-7636-4199-b466-182b242ce4da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:57.816640 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f55019d-7636-4199-b466-182b242ce4da None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:57.816916 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f55019d-7636-4199-b466-182b242ce4da None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:57.817187 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f55019d-7636-4199-b466-182b242ce4da None admin] PATH_INFO: `/v3/services/compute` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:57.817765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f55019d-7636-4199-b466-182b242ce4da None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:57.819122 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f55019d-7636-4199-b466-182b242ce4da None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:57.828819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-7f55019d-7636-4199-b466-182b242ce4da None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 31 14:44:57.828819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:57.828819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:57.828819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:57.828819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:57.828819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:57.828819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:57.828819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:57.828819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 31 14:44:57.828819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:57.828819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:57.828819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:57.828819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:57.828819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:57.828819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:57.828819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:57.828819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:57.829843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:57.829843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:57.829843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:57.829843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:57.829843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:57.829843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:57.829843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:57.829843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:57.829843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:57.829843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:57.829843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:57.829843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:57.829843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:57.829843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:57.829843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:57.829843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:57.832634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:57.832634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:57.832634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:57.832634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:57.832634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:57.832634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:57.832634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:57.832634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:57.832634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:57.832634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:57.832634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:57.832634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:57.832634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 31 14:44:57.832634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:57.832634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 383/779] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:44:57 2022] GET /identity/v3/services/compute => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:57.836109 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35df6d37-6528-4bd8-ac21-70555ca332bc None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:57.940111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35df6d37-6528-4bd8-ac21-70555ca332bc None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:57.941561 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:57.977234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed97f36-8924-45f3-957a-68e62de90f68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:57.977843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed97f36-8924-45f3-957a-68e62de90f68 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:57.978208 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed97f36-8924-45f3-957a-68e62de90f68 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:57.978762 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed97f36-8924-45f3-957a-68e62de90f68 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:57.979323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed97f36-8924-45f3-957a-68e62de90f68 None admin] RBAC: Adding query filter params (name=compute) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:57.979656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed97f36-8924-45f3-957a-68e62de90f68 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:57.981047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed97f36-8924-45f3-957a-68e62de90f68 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:57.986004 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 397/780] 10.222.0.36 () {56 vars in 1112 bytes} [Mon Oct 31 14:44:57 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:57.991495 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f55019d-7636-4199-b466-182b242ce4da None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:58.092978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f55019d-7636-4199-b466-182b242ce4da None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:58.126646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1c74e6-9268-4a76-892e-1a521a6a521c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:58.127255 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1c74e6-9268-4a76-892e-1a521a6a521c None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:58.127534 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1c74e6-9268-4a76-892e-1a521a6a521c None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:58.127834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1c74e6-9268-4a76-892e-1a521a6a521c None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:58.128462 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1c74e6-9268-4a76-892e-1a521a6a521c None admin] RBAC: Adding query filter params (type=compute) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:58.128764 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1c74e6-9268-4a76-892e-1a521a6a521c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:58.130125 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1c74e6-9268-4a76-892e-1a521a6a521c None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:58.135322 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 384/781] 10.222.0.36 () {56 vars in 1112 bytes} [Mon Oct 31 14:44:57 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:58.140821 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed97f36-8924-45f3-957a-68e62de90f68 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:58.247975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed97f36-8924-45f3-957a-68e62de90f68 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:58.283186 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92ef93c-d5f6-49dd-bbc2-61ee79907880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:58.283814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92ef93c-d5f6-49dd-bbc2-61ee79907880 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:58.284092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92ef93c-d5f6-49dd-bbc2-61ee79907880 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:58.284358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92ef93c-d5f6-49dd-bbc2-61ee79907880 None admin] PATH_INFO: `/v3/endpoints` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:58.285027 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92ef93c-d5f6-49dd-bbc2-61ee79907880 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:58.286375 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92ef93c-d5f6-49dd-bbc2-61ee79907880 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:58.311588 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 398/782] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:44:58 2022] POST /identity/v3/endpoints => generated 334 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:58.322571 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c76853-1930-4e00-bbee-062cf25b7bf0 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:58.322846 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c76853-1930-4e00-bbee-062cf25b7bf0 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:58.323119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c76853-1930-4e00-bbee-062cf25b7bf0 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:58.324559 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 385/783] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:44:58.330633 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412d23ee-e0ec-42a4-b9e5-e782ac269d00 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:58.330920 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412d23ee-e0ec-42a4-b9e5-e782ac269d00 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:58.331181 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412d23ee-e0ec-42a4-b9e5-e782ac269d00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:58.351755 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-412d23ee-e0ec-42a4-b9e5-e782ac269d00 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:58.353013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-412d23ee-e0ec-42a4-b9e5-e782ac269d00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:58.372915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:58.437001 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 399/784] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:58 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:58.709392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c76853-1930-4e00-bbee-062cf25b7bf0 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:58.725114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:58.814886 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c76853-1930-4e00-bbee-062cf25b7bf0 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:58.849215 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8348293c-af79-483b-8ffa-f4ba4c8abdcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:58.849887 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8348293c-af79-483b-8ffa-f4ba4c8abdcf None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:58.850167 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8348293c-af79-483b-8ffa-f4ba4c8abdcf None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:58.850437 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8348293c-af79-483b-8ffa-f4ba4c8abdcf None admin] PATH_INFO: `/v3/services/image` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:58.851015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8348293c-af79-483b-8ffa-f4ba4c8abdcf None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:58.852475 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8348293c-af79-483b-8ffa-f4ba4c8abdcf None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:58.861614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-8348293c-af79-483b-8ffa-f4ba4c8abdcf None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 31 14:44:58.861614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:58.861614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:44:58.861614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:44:58.861614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:44:58.861614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:44:58.861614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:44:58.861614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:58.861614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 31 14:44:58.861614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:58.861614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:44:58.861614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:58.861614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:44:58.861614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:44:58.861614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:44:58.861614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:44:58.861614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:44:58.862609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:44:58.862609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:44:58.862609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:44:58.862609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:44:58.862609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:44:58.862609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:44:58.862609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:44:58.862609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:44:58.862609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:44:58.862609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:44:58.862609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:44:58.862609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:44:58.862609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:44:58.862609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:44:58.862609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:44:58.862609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:44:58.863435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:44:58.863435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:44:58.863435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:44:58.863435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:44:58.863435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:44:58.863435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:44:58.863435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:44:58.863435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:44:58.863435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:44:58.863435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:44:58.863435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:44:58.863435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:44:58.863435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 31 14:44:58.863435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:44:58.863435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 386/785] 10.222.0.36 () {56 vars in 1105 bytes} [Mon Oct 31 14:44:58 2022] GET /identity/v3/services/image => generated 86 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:44:58.869561 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412d23ee-e0ec-42a4-b9e5-e782ac269d00 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:58.971236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412d23ee-e0ec-42a4-b9e5-e782ac269d00 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:59.005224 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049952bf-8067-4bfa-ab28-163e1b9f0644 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:59.005830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049952bf-8067-4bfa-ab28-163e1b9f0644 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:59.006110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049952bf-8067-4bfa-ab28-163e1b9f0644 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:59.006383 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049952bf-8067-4bfa-ab28-163e1b9f0644 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:59.007004 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049952bf-8067-4bfa-ab28-163e1b9f0644 None admin] RBAC: Adding query filter params (name=image) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:59.007306 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049952bf-8067-4bfa-ab28-163e1b9f0644 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:59.008710 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049952bf-8067-4bfa-ab28-163e1b9f0644 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:59.019365 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 400/786] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:58 2022] GET /identity/v3/services?name=image => generated 122 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:59.020294 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8348293c-af79-483b-8ffa-f4ba4c8abdcf None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:59.102286 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:59.126803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8348293c-af79-483b-8ffa-f4ba4c8abdcf None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:59.142137 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca6ab67-8e7f-489b-92db-516b255b56cb None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:59.142412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca6ab67-8e7f-489b-92db-516b255b56cb None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:59.142691 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca6ab67-8e7f-489b-92db-516b255b56cb None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:59.144235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 401/787] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:44:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:44:59.149162 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddef8bbc-79e3-42e7-8a89-75a4be6a6747 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:59.149440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddef8bbc-79e3-42e7-8a89-75a4be6a6747 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:59.149736 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddef8bbc-79e3-42e7-8a89-75a4be6a6747 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:59.168357 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efdc920-8869-404f-90c8-553b3b0fc381 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:59.168568 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-ddef8bbc-79e3-42e7-8a89-75a4be6a6747 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:44:59.168959 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efdc920-8869-404f-90c8-553b3b0fc381 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:59.169235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efdc920-8869-404f-90c8-553b3b0fc381 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:59.169503 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efdc920-8869-404f-90c8-553b3b0fc381 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:59.169605 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ddef8bbc-79e3-42e7-8a89-75a4be6a6747 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:44:59.170131 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efdc920-8869-404f-90c8-553b3b0fc381 None admin] RBAC: Adding query filter params (type=image) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:44:59.170431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efdc920-8869-404f-90c8-553b3b0fc381 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:59.171810 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efdc920-8869-404f-90c8-553b3b0fc381 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:59.177020 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 387/788] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:59 2022] GET /identity/v3/services?type=image => generated 350 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:44:59.181014 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:59.184013 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efdc920-8869-404f-90c8-553b3b0fc381 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:59.248793 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 402/789] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:44:59 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:44:59.288242 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efdc920-8869-404f-90c8-553b3b0fc381 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:59.325126 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5cf35c-ae3c-48eb-98b5-b7c16377b757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:59.325970 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5cf35c-ae3c-48eb-98b5-b7c16377b757 None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:59.326504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5cf35c-ae3c-48eb-98b5-b7c16377b757 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:59.326938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5cf35c-ae3c-48eb-98b5-b7c16377b757 None admin] PATH_INFO: `/v3/endpoints` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:59.327777 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5cf35c-ae3c-48eb-98b5-b7c16377b757 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:59.329306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5cf35c-ae3c-48eb-98b5-b7c16377b757 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:59.340059 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:44:59.359735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 388/790] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:44:59 2022] POST /identity/v3/endpoints => generated 327 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:44:59.523070 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddef8bbc-79e3-42e7-8a89-75a4be6a6747 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:59.625859 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddef8bbc-79e3-42e7-8a89-75a4be6a6747 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:59.665251 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc71fd4-048f-4345-ae25-20aa68610660 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:59.666476 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc71fd4-048f-4345-ae25-20aa68610660 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:59.666757 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc71fd4-048f-4345-ae25-20aa68610660 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:59.667024 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc71fd4-048f-4345-ae25-20aa68610660 None admin] PATH_INFO: `/v3/domains/default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:59.668143 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc71fd4-048f-4345-ae25-20aa68610660 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:59.669560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc71fd4-048f-4345-ae25-20aa68610660 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:59.671587 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 403/791] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:44:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:44:59.677090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5cf35c-ae3c-48eb-98b5-b7c16377b757 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:44:59.780361 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5cf35c-ae3c-48eb-98b5-b7c16377b757 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:44:59.815321 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae5805e-467e-483a-843c-441810320a10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:44:59.815958 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae5805e-467e-483a-843c-441810320a10 None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:44:59.816240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae5805e-467e-483a-843c-441810320a10 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:44:59.816512 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae5805e-467e-483a-843c-441810320a10 None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:44:59.817171 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae5805e-467e-483a-843c-441810320a10 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:44:59.818590 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae5805e-467e-483a-843c-441810320a10 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:44:59.847618 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 389/792] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:44:59 2022] POST /identity/v3/users => generated 285 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:45:00.601188 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3284ec75-aa2c-49bc-8462-aceac8b97b4e None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:00.601725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3284ec75-aa2c-49bc-8462-aceac8b97b4e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:00.601725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3284ec75-aa2c-49bc-8462-aceac8b97b4e None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:00.603670 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 404/793] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:45:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:45:00.608813 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18b3fe7-a9d0-41e8-9aa1-b275be678287 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:00.609101 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18b3fe7-a9d0-41e8-9aa1-b275be678287 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:00.609373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18b3fe7-a9d0-41e8-9aa1-b275be678287 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:00.628035 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-c18b3fe7-a9d0-41e8-9aa1-b275be678287 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:45:00.629549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-c18b3fe7-a9d0-41e8-9aa1-b275be678287 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:45:00.678098 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:00.701778 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 390/794] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:45:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:45:00.972142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3284ec75-aa2c-49bc-8462-aceac8b97b4e None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:00.981868 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:01.077699 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3284ec75-aa2c-49bc-8462-aceac8b97b4e None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:01.113656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549fb941-990f-4ab3-b8ff-be9659618c37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:01.114286 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549fb941-990f-4ab3-b8ff-be9659618c37 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:01.114565 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549fb941-990f-4ab3-b8ff-be9659618c37 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:01.114835 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549fb941-990f-4ab3-b8ff-be9659618c37 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:01.125134 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549fb941-990f-4ab3-b8ff-be9659618c37 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:45:01.130036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-549fb941-990f-4ab3-b8ff-be9659618c37 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 31 14:45:01.130036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:01.130036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:01.130036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:01.130036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:01.130036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:01.130036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:01.130036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:01.130036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:01.130036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:01.130036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:01.130036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:01.130036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:45:01.130036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:45:01.130036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:45:01.130036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:45:01.131016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:45:01.131016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:45:01.131016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:45:01.131016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:45:01.131016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:01.131016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:01.131016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:45:01.131016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:45:01.131016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:45:01.131016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:45:01.131016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:45:01.131016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:45:01.131016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:45:01.131016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:45:01.131016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:45:01.131016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:45:01.131959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:45:01.131959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:45:01.131959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:45:01.131959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:45:01.131959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:45:01.131959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:45:01.131959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:45:01.131959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:45:01.131959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:45:01.131959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:45:01.131959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 31 14:45:01.131959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:45:01.131959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 405/795] 10.222.0.36 () {56 vars in 1120 bytes} [Mon Oct 31 14:45:00 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:45:01.137499 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18b3fe7-a9d0-41e8-9aa1-b275be678287 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:01.238849 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18b3fe7-a9d0-41e8-9aa1-b275be678287 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:01.272797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993541e1-a644-4997-aa81-68c49399bef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:01.273400 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993541e1-a644-4997-aa81-68c49399bef9 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:01.273677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993541e1-a644-4997-aa81-68c49399bef9 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:01.273948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993541e1-a644-4997-aa81-68c49399bef9 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:01.274657 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993541e1-a644-4997-aa81-68c49399bef9 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:01.274955 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993541e1-a644-4997-aa81-68c49399bef9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:01.276343 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993541e1-a644-4997-aa81-68c49399bef9 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:01.284636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 391/796] 10.222.0.36 () {56 vars in 1115 bytes} [Mon Oct 31 14:45:01 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:01.291983 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549fb941-990f-4ab3-b8ff-be9659618c37 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:01.358228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:01.394066 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549fb941-990f-4ab3-b8ff-be9659618c37 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:01.427904 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d816431f-1986-4e4c-ba9e-d197e7b9554c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:01.428559 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d816431f-1986-4e4c-ba9e-d197e7b9554c None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:01.428840 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d816431f-1986-4e4c-ba9e-d197e7b9554c None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:01.429112 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d816431f-1986-4e4c-ba9e-d197e7b9554c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:01.441599 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d816431f-1986-4e4c-ba9e-d197e7b9554c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:01.443077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d816431f-1986-4e4c-ba9e-d197e7b9554c None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:01.459968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-d816431f-1986-4e4c-ba9e-d197e7b9554c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:45:01.459968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:01.459968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:01.459968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:01.459968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:01.459968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:01.459968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:01.459968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:01.459968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:01.459968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:01.459968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:01.459968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:01.459968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:45:01.459968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:45:01.459968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:45:01.459968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:45:01.461028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:01.461028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:01.461028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:45:01.461028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:45:01.461028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:45:01.461028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:45:01.461028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:45:01.461028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:45:01.461028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:45:01.461028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:45:01.461028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:45:01.461028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:45:01.461028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:45:01.461028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:45:01.461028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:45:01.461028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:45:01.461028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:45:01.461932 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:45:01.461932 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:45:01.461932 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:45:01.466272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 406/797] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:45:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:45:01.467944 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993541e1-a644-4997-aa81-68c49399bef9 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:01.515671 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:01.553489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ceacf4-dda4-4d36-96f2-dffac4d72ca2 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:01.553786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ceacf4-dda4-4d36-96f2-dffac4d72ca2 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:01.554048 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ceacf4-dda4-4d36-96f2-dffac4d72ca2 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:01.555413 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 407/798] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:45:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:45:01.561361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244327e2-7945-4c73-8e2b-ee5ac19647ff None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:01.561641 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244327e2-7945-4c73-8e2b-ee5ac19647ff None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:01.561908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244327e2-7945-4c73-8e2b-ee5ac19647ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:01.580838 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-244327e2-7945-4c73-8e2b-ee5ac19647ff None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:45:01.582041 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-244327e2-7945-4c73-8e2b-ee5ac19647ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:45:01.584268 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993541e1-a644-4997-aa81-68c49399bef9 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:01.620175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11922a45-4c2d-4a81-ab6b-dadc38347b51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:01.621143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11922a45-4c2d-4a81-ab6b-dadc38347b51 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:01.621723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11922a45-4c2d-4a81-ab6b-dadc38347b51 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:01.622190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11922a45-4c2d-4a81-ab6b-dadc38347b51 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:01.623024 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11922a45-4c2d-4a81-ab6b-dadc38347b51 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:01.623512 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11922a45-4c2d-4a81-ab6b-dadc38347b51 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:01.625080 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11922a45-4c2d-4a81-ab6b-dadc38347b51 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:01.638298 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 392/799] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:45:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:01.643511 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11922a45-4c2d-4a81-ab6b-dadc38347b51 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:01.654236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 408/800] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:45:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 31 14:45:01.745388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11922a45-4c2d-4a81-ab6b-dadc38347b51 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:01.747734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:01.781912 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d7c767-7be2-4c89-8e36-e35d33cac028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:01.782463 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d7c767-7be2-4c89-8e36-e35d33cac028 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:01.782743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d7c767-7be2-4c89-8e36-e35d33cac028 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:01.783016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d7c767-7be2-4c89-8e36-e35d33cac028 None admin] PATH_INFO: `/v3/users/nova` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:01.790118 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d7c767-7be2-4c89-8e36-e35d33cac028 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:01.792288 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d7c767-7be2-4c89-8e36-e35d33cac028 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:01.803745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-54d7c767-7be2-4c89-8e36-e35d33cac028 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 31 14:45:01.803745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:01.803745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:01.803745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:01.803745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:01.803745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:01.803745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:01.803745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:01.803745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:01.803745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:01.803745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:01.803745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:01.803745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:45:01.803745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:45:01.803745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:45:01.803745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:45:01.804723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:45:01.810593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:45:01.810593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:45:01.810593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:45:01.810593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:45:01.810593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:45:01.810593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:45:01.810593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:45:01.810593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:45:01.810593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 31 14:45:01.810593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:45:01.810593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 393/801] 10.222.0.36 () {56 vars in 1093 bytes} [Mon Oct 31 14:45:01 2022] GET /identity/v3/users/nova => generated 82 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:45:01.811212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244327e2-7945-4c73-8e2b-ee5ac19647ff None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:01.861455 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:01.914201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244327e2-7945-4c73-8e2b-ee5ac19647ff None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:01.924010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d7c767-7be2-4c89-8e36-e35d33cac028 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:01.955870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b980c462-7adb-4335-9981-4ab16e262df5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:01.956422 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b980c462-7adb-4335-9981-4ab16e262df5 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:01.956703 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b980c462-7adb-4335-9981-4ab16e262df5 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:01.956979 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b980c462-7adb-4335-9981-4ab16e262df5 None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:01.957728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b980c462-7adb-4335-9981-4ab16e262df5 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:01.958037 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b980c462-7adb-4335-9981-4ab16e262df5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:01.959508 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b980c462-7adb-4335-9981-4ab16e262df5 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:01.979101 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 409/802] 10.222.0.36 () {56 vars in 1133 bytes} [Mon Oct 31 14:45:01 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:01.984636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b980c462-7adb-4335-9981-4ab16e262df5 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:02.044062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d7c767-7be2-4c89-8e36-e35d33cac028 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:02.078339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db392ed6-69c0-4fcf-8c72-72c112adbbba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:02.078938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db392ed6-69c0-4fcf-8c72-72c112adbbba None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:02.079211 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db392ed6-69c0-4fcf-8c72-72c112adbbba None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:02.079486 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db392ed6-69c0-4fcf-8c72-72c112adbbba None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:02.080169 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db392ed6-69c0-4fcf-8c72-72c112adbbba None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:02.081514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db392ed6-69c0-4fcf-8c72-72c112adbbba None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:02.087157 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b980c462-7adb-4335-9981-4ab16e262df5 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:02.089780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.sql.core [None req-db392ed6-69c0-4fcf-8c72-72c112adbbba None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 31 14:45:02.089780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 31 14:45:02.089780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [parameters: {'id': '29203804257042109f885f73f323cb29', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 31 14:45:02.089780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=96564) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 31 14:45:02.095365 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:02.098174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-db392ed6-69c0-4fcf-8c72-72c112adbbba None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 31 14:45:02.098174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:02.098174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 31 14:45:02.098174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 31 14:45:02.098174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 31 14:45:02.098174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 31 14:45:02.098174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 31 14:45:02.098174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application result = self._query(query) Oct 31 14:45:02.098174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 31 14:45:02.098174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application conn.query(q) Oct 31 14:45:02.098174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 31 14:45:02.098174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 31 14:45:02.098174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 31 14:45:02.098174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application result.read() Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 31 14:45:02.099037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 31 14:45:02.100370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application next(self.gen) Oct 31 14:45:02.100370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 31 14:45:02.100370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application yield resource Oct 31 14:45:02.100370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 31 14:45:02.100370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application next(self.gen) Oct 31 14:45:02.100370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 31 14:45:02.100370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.session.rollback() Oct 31 14:45:02.100370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 31 14:45:02.100370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.force_reraise() Oct 31 14:45:02.100370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 31 14:45:02.100370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise self.value Oct 31 14:45:02.100370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 31 14:45:02.100370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 31 14:45:02.100370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 31 14:45:02.100370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application session.commit() Oct 31 14:45:02.100370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 31 14:45:02.100370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 31 14:45:02.101281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 31 14:45:02.101281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._prepare_impl() Oct 31 14:45:02.101281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 31 14:45:02.101281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.session.flush() Oct 31 14:45:02.101281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 31 14:45:02.101281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._flush(objects) Oct 31 14:45:02.101281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 31 14:45:02.101281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 31 14:45:02.101281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 31 14:45:02.101281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application compat.raise_( Oct 31 14:45:02.101281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 31 14:45:02.101281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception Oct 31 14:45:02.101281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 31 14:45:02.101281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application flush_context.execute() Oct 31 14:45:02.101281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 31 14:45:02.101281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rec.execute(self) Oct 31 14:45:02.101281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 31 14:45:02.102190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 31 14:45:02.102190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 31 14:45:02.102190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 31 14:45:02.102190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 31 14:45:02.102190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 31 14:45:02.102190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 31 14:45:02.102190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 31 14:45:02.102190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 31 14:45:02.102190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 31 14:45:02.102190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 31 14:45:02.102190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 31 14:45:02.102190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 31 14:45:02.102190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 31 14:45:02.102190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 31 14:45:02.102190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 31 14:45:02.103026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 31 14:45:02.103026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception Oct 31 14:45:02.103026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 31 14:45:02.103026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 31 14:45:02.103026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 31 14:45:02.103026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 31 14:45:02.103026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 31 14:45:02.103026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application result = self._query(query) Oct 31 14:45:02.103026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 31 14:45:02.103026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application conn.query(q) Oct 31 14:45:02.103026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 31 14:45:02.103026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 31 14:45:02.103026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 31 14:45:02.103026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application result.read() Oct 31 14:45:02.103026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 31 14:45:02.103026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 31 14:45:02.103026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 31 14:45:02.104711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 31 14:45:02.104711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 31 14:45:02.104711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 31 14:45:02.104711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 31 14:45:02.104711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 31 14:45:02.104711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 31 14:45:02.104711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 31 14:45:02.104711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [parameters: {'id': '29203804257042109f885f73f323cb29', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 31 14:45:02.104711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 31 14:45:02.104711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:45:02.104711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:45:02.104711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:45:02.104711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:02.104711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 31 14:45:02.104711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 31 14:45:02.105689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 31 14:45:02.105689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 31 14:45:02.105689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 31 14:45:02.105689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:45:02.105689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:45:02.105689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:45:02.105689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:02.105689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:02.105689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:02.105689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:02.105689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:02.105689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:02.105689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:02.105689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:02.105689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:02.105689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:02.105689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:02.106963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 31 14:45:02.106963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 31 14:45:02.106963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:02.106963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:02.106963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 31 14:45:02.106963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 31 14:45:02.106963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:02.106963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:02.106963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 31 14:45:02.106963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 31 14:45:02.106963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 31 14:45:02.106963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:45:02.106963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 394/803] 10.222.0.36 () {60 vars in 1137 bytes} [Mon Oct 31 14:45:01 2022] POST /identity/v3/domains => generated 190 bytes in 177 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:45:02.106963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9312dc2-93a5-4c88-a6be-f2cdccca04eb None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:02.107932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9312dc2-93a5-4c88-a6be-f2cdccca04eb None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:02.107932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9312dc2-93a5-4c88-a6be-f2cdccca04eb None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:02.107932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 395/804] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:45:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:45:02.107932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9312dc2-93a5-4c88-a6be-f2cdccca04eb None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:02.125116 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a25ce4-2322-4639-ba02-5671264c2ba9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:02.125737 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a25ce4-2322-4639-ba02-5671264c2ba9 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:02.126010 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a25ce4-2322-4639-ba02-5671264c2ba9 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:02.126286 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a25ce4-2322-4639-ba02-5671264c2ba9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:02.135740 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:02.136917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a25ce4-2322-4639-ba02-5671264c2ba9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:02.138751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a25ce4-2322-4639-ba02-5671264c2ba9 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:02.154288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-f5a25ce4-2322-4639-ba02-5671264c2ba9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:45:02.154288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:02.154288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:02.154288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:02.154288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:02.154288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:02.154288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:02.154288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:02.154288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:02.154288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:02.154288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:02.154288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:02.154288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:45:02.154288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:45:02.154288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:45:02.154288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:45:02.155440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:02.155440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:02.155440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:45:02.155440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:45:02.155440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:45:02.155440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:45:02.155440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:45:02.155440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:45:02.155440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:45:02.155440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:45:02.155440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:45:02.155440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:45:02.155440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:45:02.155440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:45:02.155440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:45:02.155440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:45:02.155440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:45:02.156668 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:45:02.156668 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:45:02.156668 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:45:02.156668 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 410/805] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:45:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:45:02.160893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0923dd7-fb38-4a30-b878-9abb0a20c358 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:02.161176 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0923dd7-fb38-4a30-b878-9abb0a20c358 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:02.161448 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0923dd7-fb38-4a30-b878-9abb0a20c358 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:02.179270 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-c0923dd7-fb38-4a30-b878-9abb0a20c358 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:45:02.180494 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-c0923dd7-fb38-4a30-b878-9abb0a20c358 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:45:02.214767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9312dc2-93a5-4c88-a6be-f2cdccca04eb None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:02.248488 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 411/806] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:45:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:45:02.248841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baedcc34-51f1-48e2-9a4a-2d0e0fb44834 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:02.249384 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0923dd7-fb38-4a30-b878-9abb0a20c358 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:02.249477 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baedcc34-51f1-48e2-9a4a-2d0e0fb44834 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:02.249737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baedcc34-51f1-48e2-9a4a-2d0e0fb44834 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:02.250007 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baedcc34-51f1-48e2-9a4a-2d0e0fb44834 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:02.265520 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baedcc34-51f1-48e2-9a4a-2d0e0fb44834 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:02.266979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baedcc34-51f1-48e2-9a4a-2d0e0fb44834 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:02.285404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-baedcc34-51f1-48e2-9a4a-2d0e0fb44834 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:45:02.285404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:02.285404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:02.285404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:02.285404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:02.285404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:02.285404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:02.285404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:02.285404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:02.285404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:02.285404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:02.285404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:02.285404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:45:02.285404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:45:02.285404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:45:02.285404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:45:02.286387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:02.286387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:02.286387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:45:02.286387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:45:02.286387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:45:02.286387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:45:02.286387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:45:02.286387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:45:02.286387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:45:02.286387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:45:02.286387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:45:02.286387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:45:02.286387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:45:02.286387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:45:02.286387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:45:02.286387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:45:02.286387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:45:02.287298 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:45:02.287298 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:45:02.287298 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:45:02.287298 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 396/807] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:45:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:45:02.293037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baedcc34-51f1-48e2-9a4a-2d0e0fb44834 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:02.338301 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:02.352523 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0923dd7-fb38-4a30-b878-9abb0a20c358 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:02.382290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:02.389313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948cd17f-b2b4-4230-840f-1dc3b24706b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:02.389916 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948cd17f-b2b4-4230-840f-1dc3b24706b6 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:02.390189 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948cd17f-b2b4-4230-840f-1dc3b24706b6 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:02.390454 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948cd17f-b2b4-4230-840f-1dc3b24706b6 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:02.391081 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948cd17f-b2b4-4230-840f-1dc3b24706b6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:02.391379 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948cd17f-b2b4-4230-840f-1dc3b24706b6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:02.392782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948cd17f-b2b4-4230-840f-1dc3b24706b6 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:02.404276 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baedcc34-51f1-48e2-9a4a-2d0e0fb44834 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:02.405018 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 412/808] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:45:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:02.410593 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948cd17f-b2b4-4230-840f-1dc3b24706b6 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:02.440170 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5607aebd-c995-4aa4-b1aa-01170859412f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:02.440760 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5607aebd-c995-4aa4-b1aa-01170859412f None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:02.441042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5607aebd-c995-4aa4-b1aa-01170859412f None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:02.441318 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5607aebd-c995-4aa4-b1aa-01170859412f None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:02.441958 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5607aebd-c995-4aa4-b1aa-01170859412f None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:02.442263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5607aebd-c995-4aa4-b1aa-01170859412f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:02.443764 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5607aebd-c995-4aa4-b1aa-01170859412f None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:02.467627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 397/809] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:45:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:02.521265 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5607aebd-c995-4aa4-b1aa-01170859412f None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:02.523687 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948cd17f-b2b4-4230-840f-1dc3b24706b6 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:02.561980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19dc742-5078-4250-b940-c2bd5362b148 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:02.562622 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19dc742-5078-4250-b940-c2bd5362b148 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:02.562901 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19dc742-5078-4250-b940-c2bd5362b148 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:02.563172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19dc742-5078-4250-b940-c2bd5362b148 None admin] PATH_INFO: `/v3/projects/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:02.570818 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19dc742-5078-4250-b940-c2bd5362b148 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:02.572440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19dc742-5078-4250-b940-c2bd5362b148 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:02.584451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-c19dc742-5078-4250-b940-c2bd5362b148 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:45:02.584451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:02.584451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:02.584451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:02.584451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:02.584451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:02.584451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:02.584451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:02.584451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:02.584451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:02.584451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:02.584451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:02.584451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:45:02.584451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:45:02.584451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:45:02.585467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:45:02.585467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:02.585467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:02.585467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:45:02.585467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:45:02.585467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:45:02.585467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:45:02.585467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:45:02.585467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:45:02.585467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:45:02.585467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:45:02.585467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:45:02.585467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:45:02.585467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:45:02.585467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:45:02.585467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:45:02.585467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:45:02.592625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:45:02.592625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:45:02.592625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:45:02.592625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:45:02.592625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:45:02.592625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:45:02.592625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:45:02.592625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:45:02.592625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 413/810] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:45:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:45:02.595791 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19dc742-5078-4250-b940-c2bd5362b148 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:02.612975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:02.650438 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5607aebd-c995-4aa4-b1aa-01170859412f None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:02.689494 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c1136f-81f0-4ab4-97a1-b27016db7d02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:02.689873 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c1136f-81f0-4ab4-97a1-b27016db7d02 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:02.690865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c1136f-81f0-4ab4-97a1-b27016db7d02 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:02.690865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c1136f-81f0-4ab4-97a1-b27016db7d02 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:02.699283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:02.704786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c1136f-81f0-4ab4-97a1-b27016db7d02 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:45:02.710374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-e9c1136f-81f0-4ab4-97a1-b27016db7d02 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:45:02.710374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:02.710374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:02.710374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:02.710374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:02.710374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:02.710374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:02.710374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:02.710374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:02.710374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:02.710374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:02.710374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:02.710374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:45:02.710374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:45:02.710374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:45:02.710374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:45:02.711325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:45:02.711325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:45:02.711325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:45:02.711325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:45:02.711325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:02.711325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:02.711325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:45:02.711325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:45:02.711325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:45:02.711325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:45:02.711325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:45:02.711325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:45:02.711325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:45:02.711325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:45:02.711325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:45:02.711325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:45:02.713015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:45:02.713015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:45:02.713015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:45:02.713015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:45:02.713015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:45:02.713015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:45:02.713015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:45:02.713015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:45:02.713015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:45:02.713015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:45:02.713015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:45:02.713015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:45:02.713015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 398/811] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:45:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:45:02.720577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c1136f-81f0-4ab4-97a1-b27016db7d02 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:02.727387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19dc742-5078-4250-b940-c2bd5362b148 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:02.769512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6435d7b5-7e98-4e9b-a289-f4bc1e0eeca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:02.770122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6435d7b5-7e98-4e9b-a289-f4bc1e0eeca1 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:02.770398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6435d7b5-7e98-4e9b-a289-f4bc1e0eeca1 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:02.770681 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6435d7b5-7e98-4e9b-a289-f4bc1e0eeca1 None admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:02.771356 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6435d7b5-7e98-4e9b-a289-f4bc1e0eeca1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:02.771697 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6435d7b5-7e98-4e9b-a289-f4bc1e0eeca1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:02.773117 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6435d7b5-7e98-4e9b-a289-f4bc1e0eeca1 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:02.785306 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 414/812] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:45:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:45:02.791788 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6435d7b5-7e98-4e9b-a289-f4bc1e0eeca1 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:02.833678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c1136f-81f0-4ab4-97a1-b27016db7d02 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:02.858063 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:02.867827 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0af173-033f-4db2-8f5d-d46108c2f8e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:02.869032 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0af173-033f-4db2-8f5d-d46108c2f8e1 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:02.869323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0af173-033f-4db2-8f5d-d46108c2f8e1 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:02.869589 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0af173-033f-4db2-8f5d-d46108c2f8e1 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:02.870257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0af173-033f-4db2-8f5d-d46108c2f8e1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:02.870561 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0af173-033f-4db2-8f5d-d46108c2f8e1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:02.871961 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0af173-033f-4db2-8f5d-d46108c2f8e1 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:02.881042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 399/813] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:45:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:02.886868 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0af173-033f-4db2-8f5d-d46108c2f8e1 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:02.891742 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6435d7b5-7e98-4e9b-a289-f4bc1e0eeca1 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:02.924819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5735d4c2-3de4-47d2-8ce5-c180949c4bce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:02.925468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5735d4c2-3de4-47d2-8ce5-c180949c4bce None admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:02.925743 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5735d4c2-3de4-47d2-8ce5-c180949c4bce None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:02.926012 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5735d4c2-3de4-47d2-8ce5-c180949c4bce None admin] PATH_INFO: `/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/c8cace029a92406f850bb0a4ca07c461/roles/2e7fb4fa015442ec912f8d8760b86724` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:02.928155 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5735d4c2-3de4-47d2-8ce5-c180949c4bce None admin] RBAC: Authorizing `identity:create_grant(project_id=4118e33b099f4cf0a3e3a46b211ba1d8, user_id=c8cace029a92406f850bb0a4ca07c461, role_id=2e7fb4fa015442ec912f8d8760b86724)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:02.929557 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5735d4c2-3de4-47d2-8ce5-c180949c4bce None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:02.942284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 415/814] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:45:02 2022] PUT /identity/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/c8cace029a92406f850bb0a4ca07c461/roles/2e7fb4fa015442ec912f8d8760b86724 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:45:02.949868 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:02.992965 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0af173-033f-4db2-8f5d-d46108c2f8e1 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:03.030431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fabbeb6-ed2c-4168-8e99-0d5fd9a7297e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:03.031076 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fabbeb6-ed2c-4168-8e99-0d5fd9a7297e None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:03.031358 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fabbeb6-ed2c-4168-8e99-0d5fd9a7297e None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:03.031664 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fabbeb6-ed2c-4168-8e99-0d5fd9a7297e None admin] PATH_INFO: `/v3/users/704743f2feec435692d5e7dd9bb5da67` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:03.047079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fabbeb6-ed2c-4168-8e99-0d5fd9a7297e None admin] RBAC: Authorizing `identity:get_user(user_id=704743f2feec435692d5e7dd9bb5da67)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:03.048576 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fabbeb6-ed2c-4168-8e99-0d5fd9a7297e None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:03.050287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 400/815] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:45:02 2022] GET /identity/v3/users/704743f2feec435692d5e7dd9bb5da67 => generated 285 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:45:03.057100 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5735d4c2-3de4-47d2-8ce5-c180949c4bce None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:03.159478 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5735d4c2-3de4-47d2-8ce5-c180949c4bce None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:03.186499 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:03.195527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53629a24-616f-43b3-b695-eb862aecea0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:03.196183 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53629a24-616f-43b3-b695-eb862aecea0f None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:03.196507 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53629a24-616f-43b3-b695-eb862aecea0f None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:03.196783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53629a24-616f-43b3-b695-eb862aecea0f None admin] PATH_INFO: `/v3/projects/b8b57ce6557149b5a859fce2525aea1b` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:03.197905 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53629a24-616f-43b3-b695-eb862aecea0f None admin] RBAC: Authorizing `identity:get_project(project_id=b8b57ce6557149b5a859fce2525aea1b)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:03.199340 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53629a24-616f-43b3-b695-eb862aecea0f None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:03.201001 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 416/816] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:45:03 2022] GET /identity/v3/projects/b8b57ce6557149b5a859fce2525aea1b => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:03.206912 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fabbeb6-ed2c-4168-8e99-0d5fd9a7297e None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:03.307324 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fabbeb6-ed2c-4168-8e99-0d5fd9a7297e None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:03.328127 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:03.343898 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c6f69a-6031-4933-9aea-d456066396e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:03.344552 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c6f69a-6031-4933-9aea-d456066396e5 None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:03.344832 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c6f69a-6031-4933-9aea-d456066396e5 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:03.345108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c6f69a-6031-4933-9aea-d456066396e5 None admin] PATH_INFO: `/v3/projects/b8b57ce6557149b5a859fce2525aea1b/users/704743f2feec435692d5e7dd9bb5da67/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:03.347020 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c6f69a-6031-4933-9aea-d456066396e5 None admin] RBAC: Authorizing `identity:create_grant(project_id=b8b57ce6557149b5a859fce2525aea1b, user_id=704743f2feec435692d5e7dd9bb5da67, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:03.348513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c6f69a-6031-4933-9aea-d456066396e5 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:03.362131 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 401/817] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:45:03 2022] PUT /identity/v3/projects/b8b57ce6557149b5a859fce2525aea1b/users/704743f2feec435692d5e7dd9bb5da67/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:45:05.120746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801ac0a8-1c87-4877-a8f1-f45740a55430 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:05.121308 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801ac0a8-1c87-4877-a8f1-f45740a55430 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:05.121308 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801ac0a8-1c87-4877-a8f1-f45740a55430 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:05.122609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 417/818] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:45:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:45:05.128134 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e769d46f-cf7c-4f37-b6ca-5d69eeef1e2b None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:05.128531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e769d46f-cf7c-4f37-b6ca-5d69eeef1e2b None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:05.128681 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e769d46f-cf7c-4f37-b6ca-5d69eeef1e2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:05.146614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-e769d46f-cf7c-4f37-b6ca-5d69eeef1e2b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:45:05.147842 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e769d46f-cf7c-4f37-b6ca-5d69eeef1e2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:45:05.214154 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 402/819] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:45:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:45:05.484298 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801ac0a8-1c87-4877-a8f1-f45740a55430 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:05.497178 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815902ab-94b4-49fd-a81b-2f03d497ccf7 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:05.497466 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815902ab-94b4-49fd-a81b-2f03d497ccf7 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:05.497733 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815902ab-94b4-49fd-a81b-2f03d497ccf7 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:05.499098 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 403/820] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:45:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:45:05.504008 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f4035d-b531-4f22-9ee7-f148d2180479 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:05.504283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f4035d-b531-4f22-9ee7-f148d2180479 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:05.504552 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f4035d-b531-4f22-9ee7-f148d2180479 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:05.523094 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-e5f4035d-b531-4f22-9ee7-f148d2180479 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:45:05.524350 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e5f4035d-b531-4f22-9ee7-f148d2180479 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:45:05.592018 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801ac0a8-1c87-4877-a8f1-f45740a55430 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:05.593235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 404/821] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:45:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 31 14:45:05.625722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05c6f59-8da5-458d-bb23-2b530e1b4b25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:05.626341 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05c6f59-8da5-458d-bb23-2b530e1b4b25 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:05.626616 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05c6f59-8da5-458d-bb23-2b530e1b4b25 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:05.626883 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05c6f59-8da5-458d-bb23-2b530e1b4b25 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:05.637393 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05c6f59-8da5-458d-bb23-2b530e1b4b25 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:45:05.642124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-c05c6f59-8da5-458d-bb23-2b530e1b4b25 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 31 14:45:05.642124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:05.642124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:05.642124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:05.642124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:05.642124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:05.642124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:05.642124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:05.642124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:05.642124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:05.642124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:05.642124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:05.642124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:45:05.642124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:45:05.642124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:45:05.642124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:45:05.643202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:45:05.643202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:45:05.643202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:45:05.643202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:45:05.643202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:05.643202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:05.643202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:45:05.643202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:45:05.643202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:45:05.643202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:45:05.643202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:45:05.643202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:45:05.643202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:45:05.643202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:45:05.643202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:45:05.643202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:45:05.644074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:45:05.644074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:45:05.644074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:45:05.644074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:45:05.644074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:45:05.644074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:45:05.644074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:45:05.644074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:45:05.644074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:45:05.644074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:45:05.644074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 31 14:45:05.644074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:45:05.644074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 418/822] 10.222.0.36 () {56 vars in 1120 bytes} [Mon Oct 31 14:45:05 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:45:05.649187 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f4035d-b531-4f22-9ee7-f148d2180479 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:05.690966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:05.753302 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f4035d-b531-4f22-9ee7-f148d2180479 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:05.787497 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0312e3e3-3a88-46f0-bdf0-2c74fb39181a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:05.788130 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0312e3e3-3a88-46f0-bdf0-2c74fb39181a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:05.788410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0312e3e3-3a88-46f0-bdf0-2c74fb39181a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:05.788682 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0312e3e3-3a88-46f0-bdf0-2c74fb39181a None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:05.789340 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0312e3e3-3a88-46f0-bdf0-2c74fb39181a None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:05.789642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0312e3e3-3a88-46f0-bdf0-2c74fb39181a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:05.791914 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0312e3e3-3a88-46f0-bdf0-2c74fb39181a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:05.800407 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 405/823] 10.222.0.36 () {56 vars in 1115 bytes} [Mon Oct 31 14:45:05 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:05.807142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05c6f59-8da5-458d-bb23-2b530e1b4b25 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:05.864510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:05.869260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0312e3e3-3a88-46f0-bdf0-2c74fb39181a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:05.909690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05c6f59-8da5-458d-bb23-2b530e1b4b25 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:05.948449 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30bedd9-3608-4c31-8737-c3a977fae69f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:05.949075 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30bedd9-3608-4c31-8737-c3a977fae69f None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:05.949350 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30bedd9-3608-4c31-8737-c3a977fae69f None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:05.949618 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30bedd9-3608-4c31-8737-c3a977fae69f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:05.959781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30bedd9-3608-4c31-8737-c3a977fae69f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:05.961202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30bedd9-3608-4c31-8737-c3a977fae69f None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:05.975292 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0312e3e3-3a88-46f0-bdf0-2c74fb39181a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:05.975657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-c30bedd9-3608-4c31-8737-c3a977fae69f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:45:05.975657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:05.975657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:05.975657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:05.975657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:05.975657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:05.975657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:05.975657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:05.975657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:05.975657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:05.975657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:05.975657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:05.975657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:45:05.975657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:45:05.975657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:45:05.975657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:45:05.976740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:05.976740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:05.976740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:45:05.976740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:45:05.976740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:45:05.976740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:45:05.976740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:45:05.976740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:45:05.976740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:45:05.976740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:45:05.976740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:45:05.976740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:45:05.976740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:45:05.976740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:45:05.976740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:45:05.976740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:45:05.976740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:45:05.978569 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:45:05.978569 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:45:05.978569 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:45:05.978569 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 419/824] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:45:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:45:05.979135 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:05.983202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30bedd9-3608-4c31-8737-c3a977fae69f None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:06.014728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a38f34b-d6b7-4cb2-899d-59c6451fd4b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:06.015277 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a38f34b-d6b7-4cb2-899d-59c6451fd4b0 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:06.015583 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a38f34b-d6b7-4cb2-899d-59c6451fd4b0 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:06.015855 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a38f34b-d6b7-4cb2-899d-59c6451fd4b0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:06.026260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a38f34b-d6b7-4cb2-899d-59c6451fd4b0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:45:06.031044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-2a38f34b-d6b7-4cb2-899d-59c6451fd4b0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:45:06.031044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:06.031044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:06.031044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:06.031044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:06.031044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:06.031044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:06.031044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:06.031044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:06.031044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:06.031044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:06.031044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:06.031044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:45:06.031044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:45:06.031044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:45:06.031044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:45:06.032011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:45:06.032011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:45:06.032011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:45:06.032011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:45:06.032011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:06.032011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:06.032011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:45:06.032011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:45:06.032011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:45:06.032011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:45:06.032011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:45:06.032011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:45:06.032011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:45:06.032011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:45:06.032011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:45:06.032011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:45:06.032834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:45:06.032834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:45:06.032834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:45:06.032834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:45:06.032834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:45:06.032834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:45:06.032834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:45:06.032834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:45:06.032834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:45:06.032834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:45:06.032834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:45:06.032834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:45:06.032834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 406/825] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:45:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:45:06.038051 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a38f34b-d6b7-4cb2-899d-59c6451fd4b0 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:06.080509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30bedd9-3608-4c31-8737-c3a977fae69f None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:06.089722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:06.115443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2bce4f-2317-4877-9c71-28a34f4ea0e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:06.116147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2bce4f-2317-4877-9c71-28a34f4ea0e7 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:06.116436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2bce4f-2317-4877-9c71-28a34f4ea0e7 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:06.116705 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2bce4f-2317-4877-9c71-28a34f4ea0e7 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:06.117336 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2bce4f-2317-4877-9c71-28a34f4ea0e7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:06.117638 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2bce4f-2317-4877-9c71-28a34f4ea0e7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:06.119020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2bce4f-2317-4877-9c71-28a34f4ea0e7 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:06.130642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 420/826] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:45:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:45:06.136637 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2bce4f-2317-4877-9c71-28a34f4ea0e7 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:06.136849 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a38f34b-d6b7-4cb2-899d-59c6451fd4b0 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:06.170147 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fb34fa-484a-480c-9578-455e5cd74e49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:06.170744 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fb34fa-484a-480c-9578-455e5cd74e49 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:06.171018 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fb34fa-484a-480c-9578-455e5cd74e49 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:06.171293 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fb34fa-484a-480c-9578-455e5cd74e49 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:06.171983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fb34fa-484a-480c-9578-455e5cd74e49 None admin] RBAC: Adding query filter params (name=admin) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:06.172287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fb34fa-484a-480c-9578-455e5cd74e49 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:06.173648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fb34fa-484a-480c-9578-455e5cd74e49 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:06.183788 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 407/827] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:45:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:45:06.189582 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fb34fa-484a-480c-9578-455e5cd74e49 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:06.214967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:06.234818 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2bce4f-2317-4877-9c71-28a34f4ea0e7 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:06.269362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618c3e2a-f2a4-4328-91bf-e8c3a9b4fe37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:06.269919 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618c3e2a-f2a4-4328-91bf-e8c3a9b4fe37 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:06.270195 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618c3e2a-f2a4-4328-91bf-e8c3a9b4fe37 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:06.270462 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618c3e2a-f2a4-4328-91bf-e8c3a9b4fe37 None admin] PATH_INFO: `/v3/users/nova` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:06.277304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618c3e2a-f2a4-4328-91bf-e8c3a9b4fe37 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:06.278726 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618c3e2a-f2a4-4328-91bf-e8c3a9b4fe37 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:06.288936 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fb34fa-484a-480c-9578-455e5cd74e49 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:06.290819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-618c3e2a-f2a4-4328-91bf-e8c3a9b4fe37 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 31 14:45:06.290819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:06.290819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:06.290819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:06.290819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:06.290819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:06.290819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:06.290819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:06.290819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:06.290819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:06.290819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:06.290819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:06.290819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:45:06.290819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:45:06.290819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:45:06.290819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:45:06.291773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:45:06.292767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:45:06.292767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:45:06.292767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:45:06.292767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:45:06.292767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:45:06.292767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:45:06.292767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:45:06.292767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:45:06.292767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 31 14:45:06.292767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:45:06.292767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 421/828] 10.222.0.36 () {56 vars in 1093 bytes} [Mon Oct 31 14:45:06 2022] GET /identity/v3/users/nova => generated 82 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:45:06.297628 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618c3e2a-f2a4-4328-91bf-e8c3a9b4fe37 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:06.324423 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327922c1-e636-4b2c-b2f8-529794712f24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:06.325047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327922c1-e636-4b2c-b2f8-529794712f24 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:06.325328 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327922c1-e636-4b2c-b2f8-529794712f24 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:06.325601 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327922c1-e636-4b2c-b2f8-529794712f24 None admin] PATH_INFO: `/v3/users/704743f2feec435692d5e7dd9bb5da67` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:06.325767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:06.327217 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327922c1-e636-4b2c-b2f8-529794712f24 None admin] RBAC: Authorizing `identity:get_user(user_id=704743f2feec435692d5e7dd9bb5da67)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:06.328963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327922c1-e636-4b2c-b2f8-529794712f24 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:06.330553 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 408/829] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:45:06 2022] GET /identity/v3/users/704743f2feec435692d5e7dd9bb5da67 => generated 285 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:45:06.336133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327922c1-e636-4b2c-b2f8-529794712f24 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:06.400802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618c3e2a-f2a4-4328-91bf-e8c3a9b4fe37 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:06.428205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:06.434100 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93ee6a6-fa55-4074-8e8d-7c00d159e747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:06.434707 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93ee6a6-fa55-4074-8e8d-7c00d159e747 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:06.434985 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93ee6a6-fa55-4074-8e8d-7c00d159e747 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:06.435265 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93ee6a6-fa55-4074-8e8d-7c00d159e747 None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:06.436019 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93ee6a6-fa55-4074-8e8d-7c00d159e747 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:06.436327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93ee6a6-fa55-4074-8e8d-7c00d159e747 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:06.437747 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93ee6a6-fa55-4074-8e8d-7c00d159e747 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:06.438001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327922c1-e636-4b2c-b2f8-529794712f24 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:06.455354 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 422/830] 10.222.0.36 () {56 vars in 1133 bytes} [Mon Oct 31 14:45:06 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:45:06.461125 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93ee6a6-fa55-4074-8e8d-7c00d159e747 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:06.473476 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c154d8b-0c47-44dd-9381-7af8c37d9fc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:06.474071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c154d8b-0c47-44dd-9381-7af8c37d9fc5 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:06.474343 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c154d8b-0c47-44dd-9381-7af8c37d9fc5 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:06.474605 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c154d8b-0c47-44dd-9381-7af8c37d9fc5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:06.545234 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c154d8b-0c47-44dd-9381-7af8c37d9fc5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:06.546676 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c154d8b-0c47-44dd-9381-7af8c37d9fc5 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:06.547596 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:06.562702 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93ee6a6-fa55-4074-8e8d-7c00d159e747 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:06.597407 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483898c0-e1cc-4d6d-98ea-c220beec0ad9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:06.598011 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483898c0-e1cc-4d6d-98ea-c220beec0ad9 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:06.598284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483898c0-e1cc-4d6d-98ea-c220beec0ad9 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:06.598551 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483898c0-e1cc-4d6d-98ea-c220beec0ad9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:06.635163 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:06.645583 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 409/831] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:45:06 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 31 14:45:06.651994 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c154d8b-0c47-44dd-9381-7af8c37d9fc5 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:06.665681 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483898c0-e1cc-4d6d-98ea-c220beec0ad9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:06.667129 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483898c0-e1cc-4d6d-98ea-c220beec0ad9 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:06.748015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c154d8b-0c47-44dd-9381-7af8c37d9fc5 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:06.748940 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:06.765958 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 423/832] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:45:06 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 31 14:45:06.772079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483898c0-e1cc-4d6d-98ea-c220beec0ad9 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:06.782780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707f5138-2f82-4eca-9eeb-af5f6d40998a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:06.783466 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707f5138-2f82-4eca-9eeb-af5f6d40998a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:06.783723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707f5138-2f82-4eca-9eeb-af5f6d40998a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:06.784002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707f5138-2f82-4eca-9eeb-af5f6d40998a None admin] PATH_INFO: `/v3/projects/b8b57ce6557149b5a859fce2525aea1b` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:06.785123 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707f5138-2f82-4eca-9eeb-af5f6d40998a None admin] RBAC: Authorizing `identity:get_project(project_id=b8b57ce6557149b5a859fce2525aea1b)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:06.786559 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707f5138-2f82-4eca-9eeb-af5f6d40998a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:06.788222 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 410/833] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:45:06 2022] GET /identity/v3/projects/b8b57ce6557149b5a859fce2525aea1b => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:06.794328 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707f5138-2f82-4eca-9eeb-af5f6d40998a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:06.853166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:06.872272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483898c0-e1cc-4d6d-98ea-c220beec0ad9 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:06.899507 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707f5138-2f82-4eca-9eeb-af5f6d40998a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:06.907541 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6caacf0c-7633-4826-99eb-c8a126468a40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:06.908197 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6caacf0c-7633-4826-99eb-c8a126468a40 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:06.908477 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6caacf0c-7633-4826-99eb-c8a126468a40 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:06.908746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6caacf0c-7633-4826-99eb-c8a126468a40 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:06.920354 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6caacf0c-7633-4826-99eb-c8a126468a40 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:06.921787 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6caacf0c-7633-4826-99eb-c8a126468a40 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:06.936825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-6caacf0c-7633-4826-99eb-c8a126468a40 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:45:06.936825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:06.936825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:06.936825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:06.936825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:06.936825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:06.936825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:06.936825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:06.936825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:06.936825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:06.936825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:06.936825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:06.936825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:45:06.936825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:45:06.936825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:45:06.936825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:45:06.937922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:06.937922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:06.937922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:45:06.937922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:45:06.937922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:45:06.937922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:45:06.937922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:45:06.937922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:45:06.937922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:45:06.937922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:45:06.937922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:45:06.937922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:45:06.937922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:45:06.937922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:45:06.937922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:45:06.937922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:45:06.937922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:45:06.938834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:45:06.938834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:45:06.938834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:45:06.939000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75770921-036e-428b-975b-b61f067a3a60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:06.939000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75770921-036e-428b-975b-b61f067a3a60 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:06.939000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75770921-036e-428b-975b-b61f067a3a60 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:06.939211 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 424/834] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:45:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:45:06.939277 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75770921-036e-428b-975b-b61f067a3a60 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:06.939355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75770921-036e-428b-975b-b61f067a3a60 None admin] RBAC: Adding query filter params (role.id=57572a37675d4c418e13794a7b8f3a5d, scope.project.id=b8b57ce6557149b5a859fce2525aea1b, user.id=704743f2feec435692d5e7dd9bb5da67) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:06.939681 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75770921-036e-428b-975b-b61f067a3a60 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:06.941039 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75770921-036e-428b-975b-b61f067a3a60 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:06.943708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6caacf0c-7633-4826-99eb-c8a126468a40 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:06.947456 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 411/835] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:45:06 2022] GET /identity/v3/role_assignments?user.id=704743f2feec435692d5e7dd9bb5da67&scope.project.id=b8b57ce6557149b5a859fce2525aea1b&role.id=57572a37675d4c418e13794a7b8f3a5d => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:45:06.988237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:07.059188 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6caacf0c-7633-4826-99eb-c8a126468a40 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:07.097814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ce1e74-f66d-432d-a1b2-c7ad08ae56b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:07.098851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ce1e74-f66d-432d-a1b2-c7ad08ae56b3 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:07.099351 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ce1e74-f66d-432d-a1b2-c7ad08ae56b3 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:07.099862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ce1e74-f66d-432d-a1b2-c7ad08ae56b3 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:07.101044 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ce1e74-f66d-432d-a1b2-c7ad08ae56b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:07.101603 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ce1e74-f66d-432d-a1b2-c7ad08ae56b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:07.103821 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ce1e74-f66d-432d-a1b2-c7ad08ae56b3 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:07.119680 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 425/836] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:45:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:45:07.126110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75770921-036e-428b-975b-b61f067a3a60 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:07.254570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75770921-036e-428b-975b-b61f067a3a60 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:07.283402 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:07.298139 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b23903-e876-492b-9b9f-c473bffb569c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:07.299588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b23903-e876-492b-9b9f-c473bffb569c None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:07.300530 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b23903-e876-492b-9b9f-c473bffb569c None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:07.301343 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b23903-e876-492b-9b9f-c473bffb569c None admin] PATH_INFO: `/v3/projects/service` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:07.311142 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b23903-e876-492b-9b9f-c473bffb569c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:07.312565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b23903-e876-492b-9b9f-c473bffb569c None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:07.328896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-28b23903-e876-492b-9b9f-c473bffb569c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:45:07.328896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:07.328896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:07.328896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:07.328896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:07.328896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:07.328896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:07.328896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:07.328896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:07.328896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:07.328896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:07.328896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:07.328896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:45:07.328896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:45:07.328896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:45:07.329871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:45:07.329871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:07.329871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:07.329871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:45:07.329871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:45:07.329871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:45:07.329871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:45:07.329871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:45:07.329871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:45:07.329871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:45:07.329871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:45:07.329871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:45:07.329871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:45:07.329871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:45:07.329871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:45:07.329871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:45:07.329871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:45:07.330765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:45:07.330765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:45:07.330765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:45:07.330765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:45:07.330765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:45:07.330765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:45:07.330765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:45:07.330765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:45:07.330765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 412/837] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:45:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:45:07.336317 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ce1e74-f66d-432d-a1b2-c7ad08ae56b3 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:07.438324 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:07.443610 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ce1e74-f66d-432d-a1b2-c7ad08ae56b3 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:07.481185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449da2a3-3cee-4b5f-b52e-0d628a74dd9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:07.481789 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449da2a3-3cee-4b5f-b52e-0d628a74dd9e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:07.482063 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449da2a3-3cee-4b5f-b52e-0d628a74dd9e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:07.482329 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449da2a3-3cee-4b5f-b52e-0d628a74dd9e None admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:07.483400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449da2a3-3cee-4b5f-b52e-0d628a74dd9e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:07.483731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449da2a3-3cee-4b5f-b52e-0d628a74dd9e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:07.485152 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449da2a3-3cee-4b5f-b52e-0d628a74dd9e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:07.500037 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 426/838] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:45:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:07.506884 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b23903-e876-492b-9b9f-c473bffb569c None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:07.627417 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b23903-e876-492b-9b9f-c473bffb569c None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:07.668265 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638e9c17-41f3-450d-862f-175c39d7fa12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:07.669052 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638e9c17-41f3-450d-862f-175c39d7fa12 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:07.669220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638e9c17-41f3-450d-862f-175c39d7fa12 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:07.669400 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638e9c17-41f3-450d-862f-175c39d7fa12 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:07.670127 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638e9c17-41f3-450d-862f-175c39d7fa12 None admin] RBAC: Adding query filter params (role.id=2e7fb4fa015442ec912f8d8760b86724, scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8, user.id=c8cace029a92406f850bb0a4ca07c461) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:07.670433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638e9c17-41f3-450d-862f-175c39d7fa12 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:07.671911 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638e9c17-41f3-450d-862f-175c39d7fa12 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:07.677999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 413/839] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:45:07 2022] GET /identity/v3/role_assignments?user.id=c8cace029a92406f850bb0a4ca07c461&scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8&role.id=2e7fb4fa015442ec912f8d8760b86724 => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:09.236489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3826893-34ed-4621-b9c7-7cabf239aaa0 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:09.237020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3826893-34ed-4621-b9c7-7cabf239aaa0 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:09.237020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3826893-34ed-4621-b9c7-7cabf239aaa0 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:09.238354 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 427/840] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:45:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:45:09.243963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343e6999-9fed-4e94-bcdf-783df3d2d0ac None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:09.244369 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343e6999-9fed-4e94-bcdf-783df3d2d0ac None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:09.244520 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343e6999-9fed-4e94-bcdf-783df3d2d0ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:09.263303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-343e6999-9fed-4e94-bcdf-783df3d2d0ac None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:45:09.264703 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-343e6999-9fed-4e94-bcdf-783df3d2d0ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:45:09.306785 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:09.330109 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 414/841] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:45:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:45:09.596535 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3826893-34ed-4621-b9c7-7cabf239aaa0 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:09.695348 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3826893-34ed-4621-b9c7-7cabf239aaa0 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:09.727946 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e487aaa7-d2bb-4998-a9a6-355b0cefcd25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:09.728550 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e487aaa7-d2bb-4998-a9a6-355b0cefcd25 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:09.728827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e487aaa7-d2bb-4998-a9a6-355b0cefcd25 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:09.729095 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e487aaa7-d2bb-4998-a9a6-355b0cefcd25 None admin] PATH_INFO: `/v3/domains/default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:09.730196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e487aaa7-d2bb-4998-a9a6-355b0cefcd25 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:09.731630 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e487aaa7-d2bb-4998-a9a6-355b0cefcd25 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:09.733171 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 428/842] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:45:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:09.738790 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343e6999-9fed-4e94-bcdf-783df3d2d0ac None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:09.834947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343e6999-9fed-4e94-bcdf-783df3d2d0ac None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:09.869413 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afc962a-c083-4a23-a861-69c3250e656e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:09.870007 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afc962a-c083-4a23-a861-69c3250e656e None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:09.870288 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afc962a-c083-4a23-a861-69c3250e656e None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:09.870563 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afc962a-c083-4a23-a861-69c3250e656e None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:09.871220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afc962a-c083-4a23-a861-69c3250e656e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:09.872680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afc962a-c083-4a23-a861-69c3250e656e None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:09.892791 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 415/843] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:45:09 2022] POST /identity/v3/users => generated 286 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:45:12.069003 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944074cc-7d03-45a0-bab1-8e7ac3b89e26 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:12.069547 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944074cc-7d03-45a0-bab1-8e7ac3b89e26 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:12.069547 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944074cc-7d03-45a0-bab1-8e7ac3b89e26 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:12.071491 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 429/844] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:45:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:45:12.076702 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ac84f4-7332-4587-94aa-0b844e84d9a6 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:12.077093 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ac84f4-7332-4587-94aa-0b844e84d9a6 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:12.077248 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ac84f4-7332-4587-94aa-0b844e84d9a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:12.096015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-e1ac84f4-7332-4587-94aa-0b844e84d9a6 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:45:12.097200 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e1ac84f4-7332-4587-94aa-0b844e84d9a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:45:12.123659 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:12.163145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 416/845] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:45:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:45:12.431872 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944074cc-7d03-45a0-bab1-8e7ac3b89e26 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:12.459115 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:12.530063 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944074cc-7d03-45a0-bab1-8e7ac3b89e26 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:12.563809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90815097-9d30-4661-be18-f20ba3d03427 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:12.564419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90815097-9d30-4661-be18-f20ba3d03427 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:12.564693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90815097-9d30-4661-be18-f20ba3d03427 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:12.564966 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90815097-9d30-4661-be18-f20ba3d03427 None admin] PATH_INFO: `/v3/roles/addf152a48404c27b7d71386e81b8f76` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:12.566327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90815097-9d30-4661-be18-f20ba3d03427 None admin] RBAC: Authorizing `identity:get_role(role_id=addf152a48404c27b7d71386e81b8f76)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:12.567702 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90815097-9d30-4661-be18-f20ba3d03427 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:12.568996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 430/846] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:45:12 2022] GET /identity/v3/roles/addf152a48404c27b7d71386e81b8f76 => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:12.575017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ac84f4-7332-4587-94aa-0b844e84d9a6 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:12.671055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ac84f4-7332-4587-94aa-0b844e84d9a6 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:12.704451 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a79d39-f8f1-4b18-829b-b744e59b8741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:12.705006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a79d39-f8f1-4b18-829b-b744e59b8741 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:12.705284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a79d39-f8f1-4b18-829b-b744e59b8741 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:12.705557 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a79d39-f8f1-4b18-829b-b744e59b8741 None admin] PATH_INFO: `/v3/users/ba47823f0d024f97ba96b7b0b5da1d08` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:12.718556 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a79d39-f8f1-4b18-829b-b744e59b8741 None admin] RBAC: Authorizing `identity:get_user(user_id=ba47823f0d024f97ba96b7b0b5da1d08)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:12.720020 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a79d39-f8f1-4b18-829b-b744e59b8741 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:12.721571 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 417/847] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:45:12 2022] GET /identity/v3/users/ba47823f0d024f97ba96b7b0b5da1d08 => generated 286 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:12.727901 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90815097-9d30-4661-be18-f20ba3d03427 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:12.816530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:12.823894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90815097-9d30-4661-be18-f20ba3d03427 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:12.856763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f2df67-c900-47ac-b62c-f238d92d0137 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:12.857374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f2df67-c900-47ac-b62c-f238d92d0137 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:12.857647 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f2df67-c900-47ac-b62c-f238d92d0137 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:12.857916 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f2df67-c900-47ac-b62c-f238d92d0137 None admin] PATH_INFO: `/v3/projects/b8b57ce6557149b5a859fce2525aea1b` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:12.859007 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f2df67-c900-47ac-b62c-f238d92d0137 None admin] RBAC: Authorizing `identity:get_project(project_id=b8b57ce6557149b5a859fce2525aea1b)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:12.860452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f2df67-c900-47ac-b62c-f238d92d0137 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:12.862052 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 431/848] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:45:12 2022] GET /identity/v3/projects/b8b57ce6557149b5a859fce2525aea1b => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:12.868510 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a79d39-f8f1-4b18-829b-b744e59b8741 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:12.892615 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:12.966785 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a79d39-f8f1-4b18-829b-b744e59b8741 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:12.999863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b6313f-e484-4060-926f-0b16919fad42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:13.000504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b6313f-e484-4060-926f-0b16919fad42 None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:13.000781 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b6313f-e484-4060-926f-0b16919fad42 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:13.001054 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b6313f-e484-4060-926f-0b16919fad42 None admin] PATH_INFO: `/v3/projects/b8b57ce6557149b5a859fce2525aea1b/users/ba47823f0d024f97ba96b7b0b5da1d08/roles/addf152a48404c27b7d71386e81b8f76` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:13.003156 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b6313f-e484-4060-926f-0b16919fad42 None admin] RBAC: Authorizing `identity:create_grant(project_id=b8b57ce6557149b5a859fce2525aea1b, user_id=ba47823f0d024f97ba96b7b0b5da1d08, role_id=addf152a48404c27b7d71386e81b8f76)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:13.004597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b6313f-e484-4060-926f-0b16919fad42 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:13.018191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 418/849] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:45:12 2022] PUT /identity/v3/projects/b8b57ce6557149b5a859fce2525aea1b/users/ba47823f0d024f97ba96b7b0b5da1d08/roles/addf152a48404c27b7d71386e81b8f76 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:45:15.127529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfefecb-765f-4e40-bf39-5313b01409bb None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:15.128079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfefecb-765f-4e40-bf39-5313b01409bb None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:15.128079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfefecb-765f-4e40-bf39-5313b01409bb None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:15.129418 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 432/850] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:45:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:45:15.134600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6108304-431c-43a1-98ac-4d2567b89f00 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:15.135003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6108304-431c-43a1-98ac-4d2567b89f00 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:15.135153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6108304-431c-43a1-98ac-4d2567b89f00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:15.153670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-d6108304-431c-43a1-98ac-4d2567b89f00 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:45:15.154862 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d6108304-431c-43a1-98ac-4d2567b89f00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:45:15.220399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 419/851] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:45:15 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:45:15.488123 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfefecb-765f-4e40-bf39-5313b01409bb None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:15.588397 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfefecb-765f-4e40-bf39-5313b01409bb None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:15.619948 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb548480-6f6f-4fe1-aaf3-4264b9ea2817 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:15.620555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb548480-6f6f-4fe1-aaf3-4264b9ea2817 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:15.620829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb548480-6f6f-4fe1-aaf3-4264b9ea2817 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:15.621097 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb548480-6f6f-4fe1-aaf3-4264b9ea2817 None admin] PATH_INFO: `/v3/roles/addf152a48404c27b7d71386e81b8f76` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:15.622458 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb548480-6f6f-4fe1-aaf3-4264b9ea2817 None admin] RBAC: Authorizing `identity:get_role(role_id=addf152a48404c27b7d71386e81b8f76)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:15.623827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb548480-6f6f-4fe1-aaf3-4264b9ea2817 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:15.625110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 433/852] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:45:15 2022] GET /identity/v3/roles/addf152a48404c27b7d71386e81b8f76 => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:15.630885 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6108304-431c-43a1-98ac-4d2567b89f00 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:15.645141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:15.728906 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6108304-431c-43a1-98ac-4d2567b89f00 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:15.761898 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b952de7-508a-4a96-817e-76f817af66c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:15.762457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b952de7-508a-4a96-817e-76f817af66c0 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:15.762738 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b952de7-508a-4a96-817e-76f817af66c0 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:15.763008 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b952de7-508a-4a96-817e-76f817af66c0 None admin] PATH_INFO: `/v3/users/ba47823f0d024f97ba96b7b0b5da1d08` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:15.764137 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b952de7-508a-4a96-817e-76f817af66c0 None admin] RBAC: Authorizing `identity:get_user(user_id=ba47823f0d024f97ba96b7b0b5da1d08)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:15.765568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b952de7-508a-4a96-817e-76f817af66c0 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:15.767114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 420/853] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:45:15 2022] GET /identity/v3/users/ba47823f0d024f97ba96b7b0b5da1d08 => generated 286 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:15.772787 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb548480-6f6f-4fe1-aaf3-4264b9ea2817 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:15.802804 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:15.870431 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb548480-6f6f-4fe1-aaf3-4264b9ea2817 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:15.902358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d971216a-dd0f-41a2-bef6-5f38af0b1c6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:15.902941 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d971216a-dd0f-41a2-bef6-5f38af0b1c6c None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:15.903217 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d971216a-dd0f-41a2-bef6-5f38af0b1c6c None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:15.903485 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d971216a-dd0f-41a2-bef6-5f38af0b1c6c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:15.972178 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d971216a-dd0f-41a2-bef6-5f38af0b1c6c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:15.973937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d971216a-dd0f-41a2-bef6-5f38af0b1c6c None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:16.002738 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:16.068590 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 434/854] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:45:15 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 31 14:45:16.075153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b952de7-508a-4a96-817e-76f817af66c0 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:16.153270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:16.171905 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b952de7-508a-4a96-817e-76f817af66c0 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:16.204639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5312955-249f-4fe5-be9a-e19f5fe94306 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:16.205246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5312955-249f-4fe5-be9a-e19f5fe94306 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:16.205523 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5312955-249f-4fe5-be9a-e19f5fe94306 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:16.205789 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5312955-249f-4fe5-be9a-e19f5fe94306 None admin] PATH_INFO: `/v3/projects/b8b57ce6557149b5a859fce2525aea1b` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:16.206869 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5312955-249f-4fe5-be9a-e19f5fe94306 None admin] RBAC: Authorizing `identity:get_project(project_id=b8b57ce6557149b5a859fce2525aea1b)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:16.208306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5312955-249f-4fe5-be9a-e19f5fe94306 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:16.209897 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 421/855] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:45:16 2022] GET /identity/v3/projects/b8b57ce6557149b5a859fce2525aea1b => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:16.216242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d971216a-dd0f-41a2-bef6-5f38af0b1c6c None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:16.311335 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d971216a-dd0f-41a2-bef6-5f38af0b1c6c None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:16.338020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:16.344241 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8718c5ec-aeb2-4ea6-9b93-ae253059bef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:16.344829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8718c5ec-aeb2-4ea6-9b93-ae253059bef9 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:16.345104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8718c5ec-aeb2-4ea6-9b93-ae253059bef9 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:16.345376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8718c5ec-aeb2-4ea6-9b93-ae253059bef9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:16.346081 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8718c5ec-aeb2-4ea6-9b93-ae253059bef9 None admin] RBAC: Adding query filter params (role.id=addf152a48404c27b7d71386e81b8f76, scope.project.id=b8b57ce6557149b5a859fce2525aea1b, user.id=ba47823f0d024f97ba96b7b0b5da1d08) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:16.346383 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8718c5ec-aeb2-4ea6-9b93-ae253059bef9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:16.347809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8718c5ec-aeb2-4ea6-9b93-ae253059bef9 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:16.353848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 435/856] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:45:16 2022] GET /identity/v3/role_assignments?user.id=ba47823f0d024f97ba96b7b0b5da1d08&scope.project.id=b8b57ce6557149b5a859fce2525aea1b&role.id=addf152a48404c27b7d71386e81b8f76 => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:18.481871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce5f667-ae13-4a71-aede-d47467d54976 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:18.482399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce5f667-ae13-4a71-aede-d47467d54976 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:18.482399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce5f667-ae13-4a71-aede-d47467d54976 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:18.483784 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 422/857] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:45:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:45:18.488910 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eff17c3-51b3-4a21-bc25-79bca5aeed8c None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:18.489276 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eff17c3-51b3-4a21-bc25-79bca5aeed8c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:18.489451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eff17c3-51b3-4a21-bc25-79bca5aeed8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:18.507916 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-2eff17c3-51b3-4a21-bc25-79bca5aeed8c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:45:18.509096 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2eff17c3-51b3-4a21-bc25-79bca5aeed8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:45:18.574195 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 436/858] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:45:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:45:18.841949 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce5f667-ae13-4a71-aede-d47467d54976 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:18.940234 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce5f667-ae13-4a71-aede-d47467d54976 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:18.974049 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57439db7-a5c7-4dea-a516-74ab4b5a06d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:18.974662 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57439db7-a5c7-4dea-a516-74ab4b5a06d0 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:18.974941 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57439db7-a5c7-4dea-a516-74ab4b5a06d0 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:18.975212 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57439db7-a5c7-4dea-a516-74ab4b5a06d0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:18.976387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57439db7-a5c7-4dea-a516-74ab4b5a06d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:18.978470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57439db7-a5c7-4dea-a516-74ab4b5a06d0 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:18.980035 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 423/859] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:45:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:18.985770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eff17c3-51b3-4a21-bc25-79bca5aeed8c None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:19.080826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eff17c3-51b3-4a21-bc25-79bca5aeed8c None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:19.104344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:19.115074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8ac9e9-108e-4282-ad8a-9d691944e022 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:19.115691 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8ac9e9-108e-4282-ad8a-9d691944e022 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:19.115970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8ac9e9-108e-4282-ad8a-9d691944e022 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:19.116240 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8ac9e9-108e-4282-ad8a-9d691944e022 None admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:19.116882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8ac9e9-108e-4282-ad8a-9d691944e022 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:19.118470 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8ac9e9-108e-4282-ad8a-9d691944e022 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:19.143000 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 437/860] 10.222.0.36 () {60 vars in 1141 bytes} [Mon Oct 31 14:45:18 2022] POST /identity/v3/projects => generated 311 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:45:21.309781 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6707ac3-efef-4532-babe-c96265820cda None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:21.310338 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6707ac3-efef-4532-babe-c96265820cda None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:21.310338 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6707ac3-efef-4532-babe-c96265820cda None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:21.311660 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 424/861] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:45:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:45:21.316952 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2892b496-652d-4cdf-9117-e2a610f9b3b6 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:21.317368 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2892b496-652d-4cdf-9117-e2a610f9b3b6 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:21.317497 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2892b496-652d-4cdf-9117-e2a610f9b3b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:21.336945 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-2892b496-652d-4cdf-9117-e2a610f9b3b6 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:45:21.338116 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2892b496-652d-4cdf-9117-e2a610f9b3b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:45:21.402193 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 438/862] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:45:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:45:21.672026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6707ac3-efef-4532-babe-c96265820cda None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:21.690320 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:21.770474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6707ac3-efef-4532-babe-c96265820cda None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:21.803342 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01177701-3a4e-46f0-a261-3323ff8a2fc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:21.803984 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01177701-3a4e-46f0-a261-3323ff8a2fc1 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:21.804873 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01177701-3a4e-46f0-a261-3323ff8a2fc1 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:21.805234 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01177701-3a4e-46f0-a261-3323ff8a2fc1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:21.806340 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01177701-3a4e-46f0-a261-3323ff8a2fc1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:21.807782 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01177701-3a4e-46f0-a261-3323ff8a2fc1 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:21.809341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 425/863] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:45:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:21.815023 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2892b496-652d-4cdf-9117-e2a610f9b3b6 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:21.909400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2892b496-652d-4cdf-9117-e2a610f9b3b6 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:21.916888 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:21.942669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0727f4db-29c0-42d1-a69b-50fd68c7f7c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:21.943203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0727f4db-29c0-42d1-a69b-50fd68c7f7c3 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:21.943481 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0727f4db-29c0-42d1-a69b-50fd68c7f7c3 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:21.943779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0727f4db-29c0-42d1-a69b-50fd68c7f7c3 None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:21.944426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0727f4db-29c0-42d1-a69b-50fd68c7f7c3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:21.945973 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0727f4db-29c0-42d1-a69b-50fd68c7f7c3 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:21.966830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 439/864] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:45:21 2022] POST /identity/v3/users => generated 286 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:45:24.118354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddea3a04-ccaa-489c-9f9f-a19212d8874e None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:24.118885 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddea3a04-ccaa-489c-9f9f-a19212d8874e None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:24.118885 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddea3a04-ccaa-489c-9f9f-a19212d8874e None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:24.120267 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 426/865] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:45:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:45:24.125467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79816115-1228-4bb6-a567-1ba4d3c13e9e None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:24.125834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79816115-1228-4bb6-a567-1ba4d3c13e9e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:24.126011 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79816115-1228-4bb6-a567-1ba4d3c13e9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:24.144210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-79816115-1228-4bb6-a567-1ba4d3c13e9e None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:45:24.145378 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-79816115-1228-4bb6-a567-1ba4d3c13e9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:45:24.209954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 440/866] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:45:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:45:24.479700 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddea3a04-ccaa-489c-9f9f-a19212d8874e None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:24.563563 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:24.580021 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddea3a04-ccaa-489c-9f9f-a19212d8874e None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:24.613216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534b58f5-c885-4077-81c8-8d4ea451c071 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:24.613826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534b58f5-c885-4077-81c8-8d4ea451c071 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:24.614105 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534b58f5-c885-4077-81c8-8d4ea451c071 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:24.614383 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534b58f5-c885-4077-81c8-8d4ea451c071 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:24.624864 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534b58f5-c885-4077-81c8-8d4ea451c071 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:45:24.629608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-534b58f5-c885-4077-81c8-8d4ea451c071 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:45:24.629608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:24.629608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:24.629608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:24.629608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:24.629608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:24.629608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:24.629608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:24.629608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:24.629608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:24.629608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:24.629608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:24.629608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:45:24.629608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:45:24.629608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:45:24.629608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:45:24.630680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:45:24.630680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:45:24.630680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:45:24.630680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:45:24.630680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:24.630680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:24.630680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:45:24.630680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:45:24.630680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:45:24.630680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:45:24.630680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:45:24.630680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:45:24.630680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:45:24.630680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:45:24.630680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:45:24.630680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:45:24.631652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:45:24.631652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:45:24.631652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:45:24.631652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:45:24.631652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:45:24.631652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:45:24.631652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:45:24.631652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:45:24.631652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:45:24.631652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:45:24.631652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:45:24.631652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:45:24.631652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 427/867] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:45:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:45:24.636762 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79816115-1228-4bb6-a567-1ba4d3c13e9e None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:24.723728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:24.733839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79816115-1228-4bb6-a567-1ba4d3c13e9e None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:24.766561 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e20808-9a9e-468e-a33e-6185bf8c3d22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:24.767151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e20808-9a9e-468e-a33e-6185bf8c3d22 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:24.767428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e20808-9a9e-468e-a33e-6185bf8c3d22 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:24.767729 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e20808-9a9e-468e-a33e-6185bf8c3d22 None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:24.768382 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e20808-9a9e-468e-a33e-6185bf8c3d22 None admin] RBAC: Adding query filter params (name=admin) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:24.768686 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e20808-9a9e-468e-a33e-6185bf8c3d22 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:24.770033 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e20808-9a9e-468e-a33e-6185bf8c3d22 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:24.778005 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 441/868] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:45:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:24.783784 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534b58f5-c885-4077-81c8-8d4ea451c071 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:24.881367 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534b58f5-c885-4077-81c8-8d4ea451c071 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:24.914784 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1b05fd-9074-4b25-a26b-95127d3f36ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:24.915394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1b05fd-9074-4b25-a26b-95127d3f36ef None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:24.915705 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1b05fd-9074-4b25-a26b-95127d3f36ef None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:24.915982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1b05fd-9074-4b25-a26b-95127d3f36ef None admin] PATH_INFO: `/v3/users/6e4d80be75584d90b3711313d466cfdb` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:24.928439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1b05fd-9074-4b25-a26b-95127d3f36ef None admin] RBAC: Authorizing `identity:get_user(user_id=6e4d80be75584d90b3711313d466cfdb)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:24.930253 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1b05fd-9074-4b25-a26b-95127d3f36ef None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:24.931863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 428/869] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:45:24 2022] GET /identity/v3/users/6e4d80be75584d90b3711313d466cfdb => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:24.937729 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e20808-9a9e-468e-a33e-6185bf8c3d22 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:25.033426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e20808-9a9e-468e-a33e-6185bf8c3d22 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:25.065355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd386f6-73f0-4a4b-8d6f-cd0311859f84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:25.065962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd386f6-73f0-4a4b-8d6f-cd0311859f84 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:25.066238 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd386f6-73f0-4a4b-8d6f-cd0311859f84 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:25.066507 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd386f6-73f0-4a4b-8d6f-cd0311859f84 None admin] PATH_INFO: `/v3/projects/86527c24b3414340985d0fd8db2886e7` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:25.067607 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd386f6-73f0-4a4b-8d6f-cd0311859f84 None admin] RBAC: Authorizing `identity:get_project(project_id=86527c24b3414340985d0fd8db2886e7)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:25.069006 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd386f6-73f0-4a4b-8d6f-cd0311859f84 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:25.070600 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 442/870] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:45:24 2022] GET /identity/v3/projects/86527c24b3414340985d0fd8db2886e7 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:25.076513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1b05fd-9074-4b25-a26b-95127d3f36ef None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:25.079368 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:25.175443 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1b05fd-9074-4b25-a26b-95127d3f36ef None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:25.208557 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ab8cf2-06d2-4479-ab28-6432b50fadad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:25.209195 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ab8cf2-06d2-4479-ab28-6432b50fadad None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:25.209474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ab8cf2-06d2-4479-ab28-6432b50fadad None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:25.209749 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ab8cf2-06d2-4479-ab28-6432b50fadad None admin] PATH_INFO: `/v3/projects/86527c24b3414340985d0fd8db2886e7/users/6e4d80be75584d90b3711313d466cfdb/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:25.212261 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ab8cf2-06d2-4479-ab28-6432b50fadad None admin] RBAC: Authorizing `identity:create_grant(project_id=86527c24b3414340985d0fd8db2886e7, user_id=6e4d80be75584d90b3711313d466cfdb, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:25.213684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ab8cf2-06d2-4479-ab28-6432b50fadad None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:25.226671 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 429/871] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:45:25 2022] PUT /identity/v3/projects/86527c24b3414340985d0fd8db2886e7/users/6e4d80be75584d90b3711313d466cfdb/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:45:27.333645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dd955e-82a2-4f22-9846-9957eece8e9b None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:27.334172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dd955e-82a2-4f22-9846-9957eece8e9b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:27.334172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dd955e-82a2-4f22-9846-9957eece8e9b None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:27.336052 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 443/872] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:45:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:45:27.341476 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674a881c-9e75-419c-8e33-89b02be73d03 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:27.341900 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674a881c-9e75-419c-8e33-89b02be73d03 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:27.342028 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674a881c-9e75-419c-8e33-89b02be73d03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:27.360378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-674a881c-9e75-419c-8e33-89b02be73d03 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:45:27.361560 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-674a881c-9e75-419c-8e33-89b02be73d03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:45:27.417291 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:27.428022 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 430/873] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:45:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 31 14:45:27.691319 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0dd955e-82a2-4f22-9846-9957eece8e9b None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:27.716685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:27.789482 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0dd955e-82a2-4f22-9846-9957eece8e9b None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:27.821929 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8c1a02-127d-4ec7-848e-7646c310b88d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:27.822538 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8c1a02-127d-4ec7-848e-7646c310b88d None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:27.822817 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8c1a02-127d-4ec7-848e-7646c310b88d None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:27.823089 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8c1a02-127d-4ec7-848e-7646c310b88d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:27.833883 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8c1a02-127d-4ec7-848e-7646c310b88d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:45:27.839196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-8e8c1a02-127d-4ec7-848e-7646c310b88d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:45:27.839196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:27.839196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:27.839196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:27.839196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:27.839196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:27.839196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:27.839196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:27.839196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:27.839196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:27.839196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:27.839196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:27.839196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:45:27.839196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:45:27.839196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:45:27.839196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:45:27.840290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:45:27.840290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:45:27.840290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:45:27.840290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:45:27.840290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:27.840290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:27.840290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:45:27.840290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:45:27.840290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:45:27.840290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:45:27.840290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:45:27.840290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:45:27.840290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:45:27.840290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:45:27.840290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:45:27.840290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:45:27.841706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:45:27.841706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:45:27.841706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:45:27.841706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:45:27.841706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:45:27.841706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:45:27.841706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:45:27.841706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:45:27.841706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:45:27.841706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:45:27.841706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:45:27.841706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:45:27.841706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 444/874] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:45:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:45:27.846571 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674a881c-9e75-419c-8e33-89b02be73d03 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:27.942273 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674a881c-9e75-419c-8e33-89b02be73d03 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:27.974867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0492ded1-4e84-4bec-92d9-8c8d72c13d26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:27.975458 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0492ded1-4e84-4bec-92d9-8c8d72c13d26 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:27.975782 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0492ded1-4e84-4bec-92d9-8c8d72c13d26 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:27.976057 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0492ded1-4e84-4bec-92d9-8c8d72c13d26 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:27.976714 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0492ded1-4e84-4bec-92d9-8c8d72c13d26 None admin] RBAC: Adding query filter params (name=admin) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:27.977015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0492ded1-4e84-4bec-92d9-8c8d72c13d26 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:27.978362 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0492ded1-4e84-4bec-92d9-8c8d72c13d26 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:27.986045 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 431/875] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:45:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:27.992577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8c1a02-127d-4ec7-848e-7646c310b88d None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:28.192863 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:28.210675 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8c1a02-127d-4ec7-848e-7646c310b88d None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:28.243491 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edc1ee6-47a9-44a5-a936-7bbabf42c2b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:28.244133 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edc1ee6-47a9-44a5-a936-7bbabf42c2b3 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:28.244408 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edc1ee6-47a9-44a5-a936-7bbabf42c2b3 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:28.244674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edc1ee6-47a9-44a5-a936-7bbabf42c2b3 None admin] PATH_INFO: `/v3/users/6e4d80be75584d90b3711313d466cfdb` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:28.245761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6edc1ee6-47a9-44a5-a936-7bbabf42c2b3 None admin] RBAC: Authorizing `identity:get_user(user_id=6e4d80be75584d90b3711313d466cfdb)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:28.247154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6edc1ee6-47a9-44a5-a936-7bbabf42c2b3 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:28.248735 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 445/876] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:45:27 2022] GET /identity/v3/users/6e4d80be75584d90b3711313d466cfdb => generated 286 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:28.254500 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0492ded1-4e84-4bec-92d9-8c8d72c13d26 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:28.309093 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:28.352989 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0492ded1-4e84-4bec-92d9-8c8d72c13d26 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:28.385356 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f25090d-f3ba-4dfb-a30f-62272769ce88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:28.385941 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f25090d-f3ba-4dfb-a30f-62272769ce88 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:28.386216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f25090d-f3ba-4dfb-a30f-62272769ce88 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:28.386487 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f25090d-f3ba-4dfb-a30f-62272769ce88 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:28.450226 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f25090d-f3ba-4dfb-a30f-62272769ce88 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:28.451611 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f25090d-f3ba-4dfb-a30f-62272769ce88 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:28.503375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:28.547936 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 432/877] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:45:28 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 31 14:45:28.554568 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edc1ee6-47a9-44a5-a936-7bbabf42c2b3 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:28.649462 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edc1ee6-47a9-44a5-a936-7bbabf42c2b3 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:28.681615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa5021f-6f7a-4b09-9a68-83ff5105117e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:28.682215 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa5021f-6f7a-4b09-9a68-83ff5105117e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:28.682493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa5021f-6f7a-4b09-9a68-83ff5105117e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:28.682759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa5021f-6f7a-4b09-9a68-83ff5105117e None admin] PATH_INFO: `/v3/projects/86527c24b3414340985d0fd8db2886e7` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:28.683877 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa5021f-6f7a-4b09-9a68-83ff5105117e None admin] RBAC: Authorizing `identity:get_project(project_id=86527c24b3414340985d0fd8db2886e7)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:28.685287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa5021f-6f7a-4b09-9a68-83ff5105117e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:28.686863 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 446/878] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:45:28 2022] GET /identity/v3/projects/86527c24b3414340985d0fd8db2886e7 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:28.692891 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f25090d-f3ba-4dfb-a30f-62272769ce88 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:28.787488 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f25090d-f3ba-4dfb-a30f-62272769ce88 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:28.820102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9310f552-e086-4460-b0dc-ac7db4675254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:28.820683 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9310f552-e086-4460-b0dc-ac7db4675254 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:28.820954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9310f552-e086-4460-b0dc-ac7db4675254 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:28.821220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9310f552-e086-4460-b0dc-ac7db4675254 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:28.821927 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9310f552-e086-4460-b0dc-ac7db4675254 None admin] RBAC: Adding query filter params (role.id=57572a37675d4c418e13794a7b8f3a5d, scope.project.id=86527c24b3414340985d0fd8db2886e7, user.id=6e4d80be75584d90b3711313d466cfdb) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:28.822228 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9310f552-e086-4460-b0dc-ac7db4675254 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:28.823660 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9310f552-e086-4460-b0dc-ac7db4675254 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:28.829156 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 433/879] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:45:28 2022] GET /identity/v3/role_assignments?user.id=6e4d80be75584d90b3711313d466cfdb&scope.project.id=86527c24b3414340985d0fd8db2886e7&role.id=57572a37675d4c418e13794a7b8f3a5d => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:30.970038 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0dbb6d-d0ba-4984-9bd5-d1d0c568ff04 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:30.970612 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0dbb6d-d0ba-4984-9bd5-d1d0c568ff04 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:30.970612 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0dbb6d-d0ba-4984-9bd5-d1d0c568ff04 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:30.971947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 447/880] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:45:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:45:30.977064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558c193b-6ba3-473b-afbe-e53db75d8799 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:30.977502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558c193b-6ba3-473b-afbe-e53db75d8799 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:30.977639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558c193b-6ba3-473b-afbe-e53db75d8799 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:30.996121 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-558c193b-6ba3-473b-afbe-e53db75d8799 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:45:30.997307 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-558c193b-6ba3-473b-afbe-e53db75d8799 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:45:31.010609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:31.062428 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 434/881] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:45:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:45:31.329530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0dbb6d-d0ba-4984-9bd5-d1d0c568ff04 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:31.348152 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:31.428433 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0dbb6d-d0ba-4984-9bd5-d1d0c568ff04 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:31.460909 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103a673a-7bdb-418c-ab8e-fc869cef1617 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:31.461497 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103a673a-7bdb-418c-ab8e-fc869cef1617 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:31.461774 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103a673a-7bdb-418c-ab8e-fc869cef1617 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:31.462044 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103a673a-7bdb-418c-ab8e-fc869cef1617 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:31.462699 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103a673a-7bdb-418c-ab8e-fc869cef1617 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:31.464081 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103a673a-7bdb-418c-ab8e-fc869cef1617 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:31.479691 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 448/882] 10.222.0.36 () {60 vars in 1138 bytes} [Mon Oct 31 14:45:31 2022] POST /identity/v3/domains => generated 267 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:45:33.638802 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c09b7e-7ccf-4d82-aab6-6e261edba98a None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:33.639367 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c09b7e-7ccf-4d82-aab6-6e261edba98a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:33.639367 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c09b7e-7ccf-4d82-aab6-6e261edba98a None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:33.640872 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 435/883] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:45:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:45:33.646174 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4814125c-f127-4955-baa7-8b07d9ffcd6d None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:33.646577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4814125c-f127-4955-baa7-8b07d9ffcd6d None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:33.646723 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4814125c-f127-4955-baa7-8b07d9ffcd6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:33.665191 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-4814125c-f127-4955-baa7-8b07d9ffcd6d None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:45:33.666361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-4814125c-f127-4955-baa7-8b07d9ffcd6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:45:33.730830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 449/884] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:45:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:45:34.000082 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c09b7e-7ccf-4d82-aab6-6e261edba98a None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:34.104486 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c09b7e-7ccf-4d82-aab6-6e261edba98a None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:34.137788 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f9135c-bb81-4159-ba92-5b1741198d51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:34.138393 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f9135c-bb81-4159-ba92-5b1741198d51 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:34.138666 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f9135c-bb81-4159-ba92-5b1741198d51 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:34.138936 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f9135c-bb81-4159-ba92-5b1741198d51 None admin] PATH_INFO: `/v3/domains/482c6da051df4e2286d59e9aaf06f641` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:34.149446 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f9135c-bb81-4159-ba92-5b1741198d51 None admin] RBAC: Authorizing `identity:get_domain(domain_id=482c6da051df4e2286d59e9aaf06f641)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:34.150869 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f9135c-bb81-4159-ba92-5b1741198d51 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:34.152452 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 436/885] 10.222.0.36 () {56 vars in 1183 bytes} [Mon Oct 31 14:45:33 2022] GET /identity/v3/domains/482c6da051df4e2286d59e9aaf06f641 => generated 267 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:34.158312 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4814125c-f127-4955-baa7-8b07d9ffcd6d None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:34.168177 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:34.255044 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4814125c-f127-4955-baa7-8b07d9ffcd6d None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:34.286576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3f567f-015a-4012-b3fb-44cbe6df64e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:34.287166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3f567f-015a-4012-b3fb-44cbe6df64e2 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:34.287440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3f567f-015a-4012-b3fb-44cbe6df64e2 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:34.288280 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3f567f-015a-4012-b3fb-44cbe6df64e2 None admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:34.288974 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3f567f-015a-4012-b3fb-44cbe6df64e2 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:34.290373 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3f567f-015a-4012-b3fb-44cbe6df64e2 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:34.316981 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 450/886] 10.222.0.36 () {60 vars in 1141 bytes} [Mon Oct 31 14:45:34 2022] POST /identity/v3/projects => generated 361 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:45:36.482230 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a1c247-a552-458c-85e5-3b0ef2c6d85a None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:36.482795 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a1c247-a552-458c-85e5-3b0ef2c6d85a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:36.482795 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a1c247-a552-458c-85e5-3b0ef2c6d85a None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:36.484203 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 437/887] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:45:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:45:36.489549 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728e61be-fb26-47e3-8b89-b2c597ae6dce None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:36.489969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728e61be-fb26-47e3-8b89-b2c597ae6dce None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:36.490102 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728e61be-fb26-47e3-8b89-b2c597ae6dce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:36.508656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-728e61be-fb26-47e3-8b89-b2c597ae6dce None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:45:36.509819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-728e61be-fb26-47e3-8b89-b2c597ae6dce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:45:36.563299 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:36.575115 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 451/888] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:45:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:45:36.846569 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a1c247-a552-458c-85e5-3b0ef2c6d85a None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:36.857772 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:36.946010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a1c247-a552-458c-85e5-3b0ef2c6d85a None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:36.979075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a3c427-84ac-466f-971b-0befd07acfa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:36.979714 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a3c427-84ac-466f-971b-0befd07acfa2 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:36.979993 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a3c427-84ac-466f-971b-0befd07acfa2 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:36.980261 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a3c427-84ac-466f-971b-0befd07acfa2 None admin] PATH_INFO: `/v3/domains/482c6da051df4e2286d59e9aaf06f641` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:36.981360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a3c427-84ac-466f-971b-0befd07acfa2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=482c6da051df4e2286d59e9aaf06f641)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:36.982784 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a3c427-84ac-466f-971b-0befd07acfa2 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:36.984358 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 438/889] 10.222.0.36 () {56 vars in 1183 bytes} [Mon Oct 31 14:45:36 2022] GET /identity/v3/domains/482c6da051df4e2286d59e9aaf06f641 => generated 267 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:36.990052 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728e61be-fb26-47e3-8b89-b2c597ae6dce None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:37.082457 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728e61be-fb26-47e3-8b89-b2c597ae6dce None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:37.114999 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e63c00-b44d-402c-bc5b-ee545dd23d39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:37.115528 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e63c00-b44d-402c-bc5b-ee545dd23d39 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:37.115832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e63c00-b44d-402c-bc5b-ee545dd23d39 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:37.116182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e63c00-b44d-402c-bc5b-ee545dd23d39 None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:37.116819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e63c00-b44d-402c-bc5b-ee545dd23d39 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:37.118216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e63c00-b44d-402c-bc5b-ee545dd23d39 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:37.137612 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 452/890] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:45:36 2022] POST /identity/v3/users => generated 311 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:45:39.295628 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedf4291-66f4-4dcd-b8fe-a6c6bb3fde73 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:39.296163 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedf4291-66f4-4dcd-b8fe-a6c6bb3fde73 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:39.296163 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedf4291-66f4-4dcd-b8fe-a6c6bb3fde73 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:39.297518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 439/891] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:45:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:45:39.302578 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc084559-1f78-4e70-aad0-2d649689525b None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:39.302991 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc084559-1f78-4e70-aad0-2d649689525b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:39.303116 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc084559-1f78-4e70-aad0-2d649689525b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:39.323296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-fc084559-1f78-4e70-aad0-2d649689525b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:45:39.324487 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-fc084559-1f78-4e70-aad0-2d649689525b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:45:39.363808 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:39.388484 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 453/892] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:45:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:45:39.655595 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedf4291-66f4-4dcd-b8fe-a6c6bb3fde73 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:39.732262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:39.755701 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedf4291-66f4-4dcd-b8fe-a6c6bb3fde73 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:39.788375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d50acd-c5a9-4d5b-a660-9c28dea72393 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:39.788980 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d50acd-c5a9-4d5b-a660-9c28dea72393 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:39.789259 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d50acd-c5a9-4d5b-a660-9c28dea72393 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:39.789530 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d50acd-c5a9-4d5b-a660-9c28dea72393 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:39.799235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d50acd-c5a9-4d5b-a660-9c28dea72393 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:45:39.804551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-91d50acd-c5a9-4d5b-a660-9c28dea72393 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:45:39.804551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:39.804551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:39.804551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:39.804551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:39.804551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:39.804551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:39.804551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:39.804551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:39.804551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:39.804551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:39.804551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:39.804551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:45:39.804551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:45:39.804551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:45:39.804551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:45:39.805720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:45:39.805720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:45:39.805720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:45:39.805720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:45:39.805720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:39.805720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:39.805720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:45:39.805720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:45:39.805720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:45:39.805720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:45:39.805720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:45:39.805720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:45:39.805720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:45:39.805720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:45:39.805720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:45:39.805720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:45:39.806826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:45:39.806826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:45:39.806826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:45:39.806826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:45:39.806826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:45:39.806826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:45:39.806826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:45:39.806826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:45:39.806826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:45:39.806826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:45:39.806826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:45:39.806826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:45:39.806826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 440/893] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:45:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:45:39.811848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc084559-1f78-4e70-aad0-2d649689525b None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:39.905359 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc084559-1f78-4e70-aad0-2d649689525b None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:39.937140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11daba39-dadc-41f5-be82-d0a7dddc49a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:39.937670 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11daba39-dadc-41f5-be82-d0a7dddc49a6 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:39.937945 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11daba39-dadc-41f5-be82-d0a7dddc49a6 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:39.938215 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11daba39-dadc-41f5-be82-d0a7dddc49a6 None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:39.938863 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11daba39-dadc-41f5-be82-d0a7dddc49a6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:39.939157 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11daba39-dadc-41f5-be82-d0a7dddc49a6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:39.940530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11daba39-dadc-41f5-be82-d0a7dddc49a6 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:39.948213 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 454/894] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:45:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:39.953935 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d50acd-c5a9-4d5b-a660-9c28dea72393 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:40.050403 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d50acd-c5a9-4d5b-a660-9c28dea72393 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:40.084951 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94411a1-5b1f-4aa0-af44-1b56a95a9548 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:40.086066 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94411a1-5b1f-4aa0-af44-1b56a95a9548 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:40.086198 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94411a1-5b1f-4aa0-af44-1b56a95a9548 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:40.086680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94411a1-5b1f-4aa0-af44-1b56a95a9548 None admin] PATH_INFO: `/v3/users/fe5be83dd38646bd9e48cf96a5918f58` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:40.090019 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:40.102522 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94411a1-5b1f-4aa0-af44-1b56a95a9548 None admin] RBAC: Authorizing `identity:get_user(user_id=fe5be83dd38646bd9e48cf96a5918f58)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:40.104044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94411a1-5b1f-4aa0-af44-1b56a95a9548 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:40.105711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 441/895] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:45:39 2022] GET /identity/v3/users/fe5be83dd38646bd9e48cf96a5918f58 => generated 311 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:40.111591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11daba39-dadc-41f5-be82-d0a7dddc49a6 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:40.152097 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:40.212181 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11daba39-dadc-41f5-be82-d0a7dddc49a6 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:40.244751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d254be3-ad79-4333-b51f-16860170466d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:40.245374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d254be3-ad79-4333-b51f-16860170466d None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:40.245654 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d254be3-ad79-4333-b51f-16860170466d None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:40.245928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d254be3-ad79-4333-b51f-16860170466d None admin] PATH_INFO: `/v3/projects/50074368d49a488abe8fbb3a9adddf79` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:40.247034 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d254be3-ad79-4333-b51f-16860170466d None admin] RBAC: Authorizing `identity:get_project(project_id=50074368d49a488abe8fbb3a9adddf79)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:40.248481 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d254be3-ad79-4333-b51f-16860170466d None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:40.250078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 455/896] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:45:40 2022] GET /identity/v3/projects/50074368d49a488abe8fbb3a9adddf79 => generated 361 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:40.255968 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94411a1-5b1f-4aa0-af44-1b56a95a9548 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:40.354821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94411a1-5b1f-4aa0-af44-1b56a95a9548 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:40.388543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a70244e-a862-4c0e-9111-fa815319034d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:40.389181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a70244e-a862-4c0e-9111-fa815319034d None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:40.389460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a70244e-a862-4c0e-9111-fa815319034d None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:40.389733 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a70244e-a862-4c0e-9111-fa815319034d None admin] PATH_INFO: `/v3/projects/50074368d49a488abe8fbb3a9adddf79/users/fe5be83dd38646bd9e48cf96a5918f58/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:40.391646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a70244e-a862-4c0e-9111-fa815319034d None admin] RBAC: Authorizing `identity:create_grant(project_id=50074368d49a488abe8fbb3a9adddf79, user_id=fe5be83dd38646bd9e48cf96a5918f58, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:40.393071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a70244e-a862-4c0e-9111-fa815319034d None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:40.406213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 442/897] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:45:40 2022] PUT /identity/v3/projects/50074368d49a488abe8fbb3a9adddf79/users/fe5be83dd38646bd9e48cf96a5918f58/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:45:42.534618 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7c66ed-68a6-40f1-a41c-d31d46137712 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:42.534994 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7c66ed-68a6-40f1-a41c-d31d46137712 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:42.535167 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7c66ed-68a6-40f1-a41c-d31d46137712 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:42.536537 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 456/898] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:45:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:45:42.542083 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578d1037-322a-4945-8768-ceda7133d38b None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:42.542509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578d1037-322a-4945-8768-ceda7133d38b None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:42.542607 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578d1037-322a-4945-8768-ceda7133d38b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:42.560306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-578d1037-322a-4945-8768-ceda7133d38b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:45:42.561479 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-578d1037-322a-4945-8768-ceda7133d38b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:45:42.609753 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:42.627692 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 443/899] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:45:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:45:42.898045 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7c66ed-68a6-40f1-a41c-d31d46137712 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:42.996045 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7c66ed-68a6-40f1-a41c-d31d46137712 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:43.002569 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:43.029258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a224fa-05a7-4539-a7ce-b7c155e28c92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:43.029872 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a224fa-05a7-4539-a7ce-b7c155e28c92 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:43.030148 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a224fa-05a7-4539-a7ce-b7c155e28c92 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:43.030420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a224fa-05a7-4539-a7ce-b7c155e28c92 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:43.040149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a224fa-05a7-4539-a7ce-b7c155e28c92 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:45:43.044899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-88a224fa-05a7-4539-a7ce-b7c155e28c92 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:45:43.044899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:45:43.044899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:45:43.044899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:45:43.044899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:45:43.044899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:45:43.044899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:45:43.044899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:45:43.044899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:45:43.044899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:45:43.044899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:45:43.044899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:45:43.044899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:45:43.044899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:45:43.044899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:45:43.044899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:45:43.045939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:45:43.045939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:45:43.045939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:45:43.045939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:45:43.045939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:45:43.045939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:45:43.045939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:45:43.045939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:45:43.045939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:45:43.045939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:45:43.045939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:45:43.045939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:45:43.045939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:45:43.045939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:45:43.045939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:45:43.045939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:45:43.046761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:45:43.046761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:45:43.046761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:45:43.046761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:45:43.046761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:45:43.046761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:45:43.046761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:45:43.046761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:45:43.046761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:45:43.046761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:45:43.046761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:45:43.046761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:45:43.046761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 457/900] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:45:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:45:43.051929 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578d1037-322a-4945-8768-ceda7133d38b None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:43.147478 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578d1037-322a-4945-8768-ceda7133d38b None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:43.180038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8b6fa4-4947-47f3-a01c-12e1b845c797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:43.180649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8b6fa4-4947-47f3-a01c-12e1b845c797 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:43.181051 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8b6fa4-4947-47f3-a01c-12e1b845c797 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:43.181357 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8b6fa4-4947-47f3-a01c-12e1b845c797 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:43.182395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8b6fa4-4947-47f3-a01c-12e1b845c797 None admin] RBAC: Adding query filter params (name=admin) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:43.182556 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8b6fa4-4947-47f3-a01c-12e1b845c797 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:43.184474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8b6fa4-4947-47f3-a01c-12e1b845c797 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:43.193190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 444/901] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:45:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:43.199660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a224fa-05a7-4539-a7ce-b7c155e28c92 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:43.296123 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a224fa-05a7-4539-a7ce-b7c155e28c92 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:43.328455 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea57df3-3fec-4bd3-8ffe-8122634f35e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:43.329064 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea57df3-3fec-4bd3-8ffe-8122634f35e5 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:43.329340 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea57df3-3fec-4bd3-8ffe-8122634f35e5 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:43.329607 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea57df3-3fec-4bd3-8ffe-8122634f35e5 None admin] PATH_INFO: `/v3/users/fe5be83dd38646bd9e48cf96a5918f58` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:43.330710 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea57df3-3fec-4bd3-8ffe-8122634f35e5 None admin] RBAC: Authorizing `identity:get_user(user_id=fe5be83dd38646bd9e48cf96a5918f58)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:43.332148 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea57df3-3fec-4bd3-8ffe-8122634f35e5 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:43.333677 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 458/902] 10.222.0.36 () {56 vars in 1177 bytes} [Mon Oct 31 14:45:43 2022] GET /identity/v3/users/fe5be83dd38646bd9e48cf96a5918f58 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:43.339469 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8b6fa4-4947-47f3-a01c-12e1b845c797 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:43.386842 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:43.438621 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8b6fa4-4947-47f3-a01c-12e1b845c797 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:43.470811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5e8575-6f86-4cf9-9e19-154e272f9123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:43.471394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5e8575-6f86-4cf9-9e19-154e272f9123 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:43.471700 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5e8575-6f86-4cf9-9e19-154e272f9123 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:43.471973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5e8575-6f86-4cf9-9e19-154e272f9123 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:43.535924 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5e8575-6f86-4cf9-9e19-154e272f9123 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:43.537898 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5e8575-6f86-4cf9-9e19-154e272f9123 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:43.580672 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:43.632844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 445/903] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:45:43 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 31 14:45:43.638720 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea57df3-3fec-4bd3-8ffe-8122634f35e5 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:43.672631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:43.736563 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea57df3-3fec-4bd3-8ffe-8122634f35e5 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:43.768127 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63003ad-1c29-4d7c-a7c3-938589e7d10e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:43.768746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63003ad-1c29-4d7c-a7c3-938589e7d10e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:43.769022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63003ad-1c29-4d7c-a7c3-938589e7d10e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:43.769294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63003ad-1c29-4d7c-a7c3-938589e7d10e None admin] PATH_INFO: `/v3/projects/50074368d49a488abe8fbb3a9adddf79` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:43.770389 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63003ad-1c29-4d7c-a7c3-938589e7d10e None admin] RBAC: Authorizing `identity:get_project(project_id=50074368d49a488abe8fbb3a9adddf79)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:43.773147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63003ad-1c29-4d7c-a7c3-938589e7d10e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:43.774754 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 459/904] 10.222.0.36 () {56 vars in 1186 bytes} [Mon Oct 31 14:45:43 2022] GET /identity/v3/projects/50074368d49a488abe8fbb3a9adddf79 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:43.780804 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5e8575-6f86-4cf9-9e19-154e272f9123 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:45:43.876855 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5e8575-6f86-4cf9-9e19-154e272f9123 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:45:43.909595 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2014991-fed7-424a-9735-f57f2b6ad646 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:45:43.910176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2014991-fed7-424a-9735-f57f2b6ad646 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:45:43.910454 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2014991-fed7-424a-9735-f57f2b6ad646 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:45:43.910723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2014991-fed7-424a-9735-f57f2b6ad646 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:45:43.911429 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2014991-fed7-424a-9735-f57f2b6ad646 None admin] RBAC: Adding query filter params (role.id=57572a37675d4c418e13794a7b8f3a5d, scope.project.id=50074368d49a488abe8fbb3a9adddf79, user.id=fe5be83dd38646bd9e48cf96a5918f58) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:45:43.911755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2014991-fed7-424a-9735-f57f2b6ad646 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:45:43.913702 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2014991-fed7-424a-9735-f57f2b6ad646 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:45:43.919229 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 446/905] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:45:43 2022] GET /identity/v3/role_assignments?user.id=fe5be83dd38646bd9e48cf96a5918f58&scope.project.id=50074368d49a488abe8fbb3a9adddf79&role.id=57572a37675d4c418e13794a7b8f3a5d => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:45:53.764148 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=49 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:45:53.916721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=95 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:39.765390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcc49a9-8aee-4f81-a708-a18ebf5e2c93 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:39.765968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcc49a9-8aee-4f81-a708-a18ebf5e2c93 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:39.765968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcc49a9-8aee-4f81-a708-a18ebf5e2c93 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:39.768221 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 460/906] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:46:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:46:39.774048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb813d8b-ba68-43db-825b-5c00c64bcdfb None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:39.774550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb813d8b-ba68-43db-825b-5c00c64bcdfb None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:39.774638 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb813d8b-ba68-43db-825b-5c00c64bcdfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:39.795814 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-fb813d8b-ba68-43db-825b-5c00c64bcdfb None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:46:39.796936 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-fb813d8b-ba68-43db-825b-5c00c64bcdfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:46:39.896190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 447/907] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:46:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:46:40.190557 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcc49a9-8aee-4f81-a708-a18ebf5e2c93 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:40.330614 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcc49a9-8aee-4f81-a708-a18ebf5e2c93 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:40.382823 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c412f918-e481-4c83-a54e-ab86b67fb6a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:40.383926 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c412f918-e481-4c83-a54e-ab86b67fb6a4 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:40.384550 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c412f918-e481-4c83-a54e-ab86b67fb6a4 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:40.385126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c412f918-e481-4c83-a54e-ab86b67fb6a4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:40.398461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c412f918-e481-4c83-a54e-ab86b67fb6a4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:40.400255 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c412f918-e481-4c83-a54e-ab86b67fb6a4 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:40.417445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-c412f918-e481-4c83-a54e-ab86b67fb6a4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:40.417445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:40.417445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:40.417445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:40.417445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:40.417445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:40.417445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:40.417445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:40.417445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:40.417445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:40.417445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:40.417445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:40.417445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:46:40.417445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:46:40.417445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:46:40.417445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:46:40.418576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:40.418576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:40.418576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:40.418576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:40.418576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:40.418576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:40.418576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:40.418576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:40.418576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:40.418576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:40.418576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:40.418576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:40.418576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:40.418576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:40.418576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:40.418576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:40.418576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:46:40.419640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:46:40.419640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:40.419640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:46:40.422084 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 461/908] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:46:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:46:40.429023 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb813d8b-ba68-43db-825b-5c00c64bcdfb None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:40.569327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb813d8b-ba68-43db-825b-5c00c64bcdfb None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:40.607532 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:40.615204 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1ab95b-bc38-40cb-90d0-c67f4d30e953 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:40.615860 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1ab95b-bc38-40cb-90d0-c67f4d30e953 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:40.616150 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1ab95b-bc38-40cb-90d0-c67f4d30e953 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:40.616405 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1ab95b-bc38-40cb-90d0-c67f4d30e953 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:40.617051 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1ab95b-bc38-40cb-90d0-c67f4d30e953 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:40.617483 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1ab95b-bc38-40cb-90d0-c67f4d30e953 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:40.619038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1ab95b-bc38-40cb-90d0-c67f4d30e953 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:40.632581 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 448/909] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:46:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:46:40.638100 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c412f918-e481-4c83-a54e-ab86b67fb6a4 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:40.711396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:40.779882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c412f918-e481-4c83-a54e-ab86b67fb6a4 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:40.822080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e3aa9c-b2a9-4ba5-a6ee-e48e33f8a858 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:40.822769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e3aa9c-b2a9-4ba5-a6ee-e48e33f8a858 None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:40.823075 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e3aa9c-b2a9-4ba5-a6ee-e48e33f8a858 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:40.823393 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e3aa9c-b2a9-4ba5-a6ee-e48e33f8a858 None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:40.824201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e3aa9c-b2a9-4ba5-a6ee-e48e33f8a858 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:40.825888 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e3aa9c-b2a9-4ba5-a6ee-e48e33f8a858 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:40.861372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 462/910] 10.222.0.36 () {60 vars in 1132 bytes} [Mon Oct 31 14:46:40 2022] POST /identity/v3/users => generated 251 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:46:43.221608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9244cd-bfec-4e6f-a024-6e055faa9111 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:43.222226 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9244cd-bfec-4e6f-a024-6e055faa9111 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:43.222226 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9244cd-bfec-4e6f-a024-6e055faa9111 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:43.223686 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 449/911] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:46:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:46:43.230015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1795b07c-60a4-4e37-b180-b4ecfa20f825 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:43.230503 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1795b07c-60a4-4e37-b180-b4ecfa20f825 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:43.230619 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1795b07c-60a4-4e37-b180-b4ecfa20f825 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:43.250772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-1795b07c-60a4-4e37-b180-b4ecfa20f825 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:46:43.252290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1795b07c-60a4-4e37-b180-b4ecfa20f825 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:46:43.332877 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 463/912] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:46:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:46:43.606974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9244cd-bfec-4e6f-a024-6e055faa9111 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:43.722948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9244cd-bfec-4e6f-a024-6e055faa9111 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:43.762135 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636f5f42-db09-4eef-9f3f-a5e91088202a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:43.762818 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636f5f42-db09-4eef-9f3f-a5e91088202a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:43.763102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636f5f42-db09-4eef-9f3f-a5e91088202a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:43.763372 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636f5f42-db09-4eef-9f3f-a5e91088202a None admin] PATH_INFO: `/v3/roles/service` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:43.773982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636f5f42-db09-4eef-9f3f-a5e91088202a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:46:43.779086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-636f5f42-db09-4eef-9f3f-a5e91088202a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:46:43.779086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:43.779086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:43.779086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:43.779086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:43.779086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:43.779086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:43.779086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:43.779086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:43.779086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:43.779086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:43.779086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:43.779086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:46:43.779086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:46:43.779086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:46:43.779086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:46:43.780353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:46:43.780353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:46:43.780353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:46:43.780353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:46:43.780353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:43.780353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:43.780353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:43.780353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:43.780353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:43.780353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:43.780353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:43.780353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:43.780353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:43.780353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:43.780353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:43.780353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:43.781181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:43.781181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:43.781181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:43.781181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:43.781181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:46:43.781181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:46:43.781181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:46:43.781181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:46:43.781181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:46:43.781181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:46:43.781181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:46:43.781181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:46:43.789212 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 450/913] 10.222.0.36 () {56 vars in 1102 bytes} [Mon Oct 31 14:46:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:46:43.789383 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1795b07c-60a4-4e37-b180-b4ecfa20f825 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:43.833706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:43.901902 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1795b07c-60a4-4e37-b180-b4ecfa20f825 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:43.939922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed622e67-0807-4b33-845d-c48fe49faadb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:43.940673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed622e67-0807-4b33-845d-c48fe49faadb None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:43.940999 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed622e67-0807-4b33-845d-c48fe49faadb None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:43.941225 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed622e67-0807-4b33-845d-c48fe49faadb None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:43.941980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed622e67-0807-4b33-845d-c48fe49faadb None admin] RBAC: Adding query filter params (name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:43.942282 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed622e67-0807-4b33-845d-c48fe49faadb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:43.943822 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed622e67-0807-4b33-845d-c48fe49faadb None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:43.957405 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 464/914] 10.222.0.36 () {56 vars in 1103 bytes} [Mon Oct 31 14:46:43 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:46:43.958678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636f5f42-db09-4eef-9f3f-a5e91088202a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:44.015358 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:44.071724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636f5f42-db09-4eef-9f3f-a5e91088202a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:44.113650 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f46db2-a7e5-443a-aed3-eb220759ad1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:44.114275 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f46db2-a7e5-443a-aed3-eb220759ad1a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:44.114552 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f46db2-a7e5-443a-aed3-eb220759ad1a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:44.114821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f46db2-a7e5-443a-aed3-eb220759ad1a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:44.124898 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f46db2-a7e5-443a-aed3-eb220759ad1a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:44.126359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f46db2-a7e5-443a-aed3-eb220759ad1a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:44.142355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-14f46db2-a7e5-443a-aed3-eb220759ad1a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:44.142355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:44.142355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:44.142355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:44.142355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:44.142355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:44.142355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:44.142355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:44.142355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:44.142355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:44.142355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:44.142355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:44.142355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:46:44.142355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:46:44.142355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:46:44.142355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:46:44.148360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:44.148360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:44.148360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:44.148360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:44.148360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:44.148360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:44.148360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:44.148360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:44.148360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:44.148360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:44.148360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:44.148360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:44.148360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:44.148360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:44.148360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:44.148360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:44.148360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:46:44.149489 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:46:44.149489 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:44.149489 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:46:44.149489 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 451/915] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:46:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:46:44.149760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed622e67-0807-4b33-845d-c48fe49faadb None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:44.265959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:44.272701 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed622e67-0807-4b33-845d-c48fe49faadb None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:44.317247 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d592b330-f125-42aa-9afb-5e5362c3bb86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:44.317247 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d592b330-f125-42aa-9afb-5e5362c3bb86 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:44.317576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d592b330-f125-42aa-9afb-5e5362c3bb86 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:44.318655 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d592b330-f125-42aa-9afb-5e5362c3bb86 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:44.318655 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d592b330-f125-42aa-9afb-5e5362c3bb86 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:44.319385 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d592b330-f125-42aa-9afb-5e5362c3bb86 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:44.321879 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d592b330-f125-42aa-9afb-5e5362c3bb86 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:44.339601 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 465/916] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:46:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:46:44.348247 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f46db2-a7e5-443a-aed3-eb220759ad1a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:44.472409 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f46db2-a7e5-443a-aed3-eb220759ad1a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:44.474317 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:44.521024 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c543d2bb-edfd-4baa-8a0d-c32f4f11808f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:44.521740 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c543d2bb-edfd-4baa-8a0d-c32f4f11808f None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:44.522074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c543d2bb-edfd-4baa-8a0d-c32f4f11808f None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:44.522399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c543d2bb-edfd-4baa-8a0d-c32f4f11808f None admin] PATH_INFO: `/v3/users/placement` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:44.538115 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c543d2bb-edfd-4baa-8a0d-c32f4f11808f None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:44.540098 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c543d2bb-edfd-4baa-8a0d-c32f4f11808f None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:44.557166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-c543d2bb-edfd-4baa-8a0d-c32f4f11808f None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 31 14:46:44.557166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:44.557166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:44.557166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:44.557166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:44.557166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:44.557166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:44.557166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:44.557166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:44.557166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:44.557166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:44.557166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:44.557166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:46:44.557166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:46:44.557166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:46:44.557166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:44.558401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:44.559619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:44.559619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:44.559619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:46:44.559619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:46:44.559619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:46:44.559619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:46:44.559619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:46:44.559619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:46:44.559619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 31 14:46:44.559619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:46:44.559619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 452/917] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:46:44 2022] GET /identity/v3/users/placement => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:46:44.565993 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d592b330-f125-42aa-9afb-5e5362c3bb86 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:44.679191 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d592b330-f125-42aa-9afb-5e5362c3bb86 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:44.713342 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6eb3dc6-1846-443b-abb2-483e9afdf4c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:44.714199 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6eb3dc6-1846-443b-abb2-483e9afdf4c3 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:44.714484 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6eb3dc6-1846-443b-abb2-483e9afdf4c3 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:44.714826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6eb3dc6-1846-443b-abb2-483e9afdf4c3 None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:44.715474 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6eb3dc6-1846-443b-abb2-483e9afdf4c3 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:44.715952 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6eb3dc6-1846-443b-abb2-483e9afdf4c3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:44.717460 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6eb3dc6-1846-443b-abb2-483e9afdf4c3 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:44.739173 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 466/918] 10.222.0.36 () {56 vars in 1143 bytes} [Mon Oct 31 14:46:44 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:46:44.745528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c543d2bb-edfd-4baa-8a0d-c32f4f11808f None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:44.860157 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c543d2bb-edfd-4baa-8a0d-c32f4f11808f None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:44.902370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e07ec8c-4eaf-4b6e-ac78-f78e72b19831 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:44.902999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e07ec8c-4eaf-4b6e-ac78-f78e72b19831 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:44.903281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e07ec8c-4eaf-4b6e-ac78-f78e72b19831 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:44.903735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e07ec8c-4eaf-4b6e-ac78-f78e72b19831 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:44.915415 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e07ec8c-4eaf-4b6e-ac78-f78e72b19831 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:44.916900 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e07ec8c-4eaf-4b6e-ac78-f78e72b19831 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:44.933221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-6e07ec8c-4eaf-4b6e-ac78-f78e72b19831 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:44.933221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:44.933221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:44.933221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:44.933221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:44.933221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:44.933221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:44.933221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:44.933221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:44.933221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:44.933221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:44.933221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:44.933221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:46:44.933221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:46:44.933221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:46:44.933221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:46:44.934329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:44.934329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:44.934329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:44.934329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:44.934329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:44.934329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:44.934329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:44.934329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:44.934329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:44.934329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:44.934329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:44.934329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:44.934329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:44.934329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:44.934329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:44.934329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:44.934329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:46:44.935669 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:46:44.935669 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:44.935669 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:46:44.935669 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 453/919] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:46:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:46:44.940248 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6eb3dc6-1846-443b-abb2-483e9afdf4c3 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:44.962629 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:45.058463 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6eb3dc6-1846-443b-abb2-483e9afdf4c3 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:45.104281 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7c1682-7f0a-4fca-82f3-c803fa95a679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:45.105283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c1682-7f0a-4fca-82f3-c803fa95a679 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:45.105627 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c1682-7f0a-4fca-82f3-c803fa95a679 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:45.105924 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c1682-7f0a-4fca-82f3-c803fa95a679 None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:45.106641 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7c1682-7f0a-4fca-82f3-c803fa95a679 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:45.107022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7c1682-7f0a-4fca-82f3-c803fa95a679 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:45.108591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7c1682-7f0a-4fca-82f3-c803fa95a679 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:45.123532 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 467/920] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:46:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:46:45.129247 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e07ec8c-4eaf-4b6e-ac78-f78e72b19831 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:45.147889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:45.262154 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e07ec8c-4eaf-4b6e-ac78-f78e72b19831 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:45.301006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ddb642-1915-4fba-8144-f33e89657366 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:45.301746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ddb642-1915-4fba-8144-f33e89657366 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:45.302032 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ddb642-1915-4fba-8144-f33e89657366 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:45.302339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ddb642-1915-4fba-8144-f33e89657366 None admin] PATH_INFO: `/v3/projects/service` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:45.312255 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ddb642-1915-4fba-8144-f33e89657366 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:45.313891 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ddb642-1915-4fba-8144-f33e89657366 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:45.328241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-62ddb642-1915-4fba-8144-f33e89657366 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:46:45.328241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:45.328241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:45.328241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:45.328241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:45.328241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:45.328241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:45.328241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:45.328241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:45.328241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:45.328241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:45.328241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:45.328241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:46:45.328241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:46:45.328241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:46:45.336047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:46:45.336047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:45.336047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:45.336047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:45.336047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:45.336047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:45.336047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:45.336047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:45.336047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:45.336047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:45.336047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:45.336047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:45.336047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:45.336047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:45.336047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:45.336047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:45.336047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:45.336964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:46:45.336964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:46:45.336964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:46:45.336964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:46:45.336964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:46:45.336964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:46:45.336964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:46:45.336964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:46:45.336964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 454/921] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:46:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:46:45.337886 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7c1682-7f0a-4fca-82f3-c803fa95a679 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:45.427415 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:45.460087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7c1682-7f0a-4fca-82f3-c803fa95a679 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:45.506626 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f08496d-6438-475c-b919-82a9a2f53fe8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:45.507259 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f08496d-6438-475c-b919-82a9a2f53fe8 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:45.507537 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f08496d-6438-475c-b919-82a9a2f53fe8 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:45.507913 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f08496d-6438-475c-b919-82a9a2f53fe8 None admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:45.508603 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f08496d-6438-475c-b919-82a9a2f53fe8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:45.508982 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f08496d-6438-475c-b919-82a9a2f53fe8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:45.510403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f08496d-6438-475c-b919-82a9a2f53fe8 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:45.529046 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 468/922] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:46:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:46:45.536180 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ddb642-1915-4fba-8144-f33e89657366 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:45.620673 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:45.648245 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ddb642-1915-4fba-8144-f33e89657366 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:45.685126 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cefaa6-5084-46f1-af8a-3df72974b9f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:45.685781 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cefaa6-5084-46f1-af8a-3df72974b9f0 None admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:45.686062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cefaa6-5084-46f1-af8a-3df72974b9f0 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:45.686363 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cefaa6-5084-46f1-af8a-3df72974b9f0 None admin] PATH_INFO: `/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/70ea2aaf529b46dabb70b71f4dcc3860/roles/88d62c66100a4d04a45e3607135bdd0c` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:45.700734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cefaa6-5084-46f1-af8a-3df72974b9f0 None admin] RBAC: Authorizing `identity:create_grant(project_id=4118e33b099f4cf0a3e3a46b211ba1d8, user_id=70ea2aaf529b46dabb70b71f4dcc3860, role_id=88d62c66100a4d04a45e3607135bdd0c)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:45.702171 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cefaa6-5084-46f1-af8a-3df72974b9f0 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:45.715098 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 455/923] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:46:45 2022] PUT /identity/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/70ea2aaf529b46dabb70b71f4dcc3860/roles/88d62c66100a4d04a45e3607135bdd0c => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:46:47.983433 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0448d8ef-d05a-4446-b1c2-c38998a28fb3 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:47.984041 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0448d8ef-d05a-4446-b1c2-c38998a28fb3 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:47.984041 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0448d8ef-d05a-4446-b1c2-c38998a28fb3 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:47.985628 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 469/924] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:46:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:46:47.991900 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d963e9c8-9d9c-498c-8b55-932214421e52 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:47.992353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d963e9c8-9d9c-498c-8b55-932214421e52 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:47.992441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d963e9c8-9d9c-498c-8b55-932214421e52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:48.012890 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-d963e9c8-9d9c-498c-8b55-932214421e52 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:46:48.014166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d963e9c8-9d9c-498c-8b55-932214421e52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:46:48.086422 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 456/925] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:46:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:46:48.385638 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0448d8ef-d05a-4446-b1c2-c38998a28fb3 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:48.500043 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0448d8ef-d05a-4446-b1c2-c38998a28fb3 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:48.540233 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57932277-bafb-4ddf-a977-d1e357067e84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:48.540982 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57932277-bafb-4ddf-a977-d1e357067e84 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:48.541276 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57932277-bafb-4ddf-a977-d1e357067e84 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:48.541543 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57932277-bafb-4ddf-a977-d1e357067e84 None admin] PATH_INFO: `/v3/roles/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:48.549748 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:48.554643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57932277-bafb-4ddf-a977-d1e357067e84 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:46:48.560119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-57932277-bafb-4ddf-a977-d1e357067e84 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:46:48.560119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:48.560119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:48.560119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:48.560119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:48.560119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:48.560119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:48.560119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:48.560119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:48.560119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:48.560119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:48.560119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:48.560119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:46:48.560119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:46:48.560119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:46:48.560119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:46:48.561237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:46:48.561237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:46:48.561237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:46:48.561237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:46:48.561237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:48.561237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:48.561237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:48.561237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:48.561237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:48.561237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:48.561237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:48.561237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:48.561237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:48.561237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:48.561237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:48.561237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:48.565110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:48.565110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:48.565110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:48.565110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:48.565110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:46:48.565110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:46:48.565110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:46:48.565110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:46:48.565110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:46:48.565110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:46:48.565110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 31 14:46:48.565110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:46:48.565110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 470/926] 10.222.0.36 () {56 vars in 1102 bytes} [Mon Oct 31 14:46:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:46:48.568108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d963e9c8-9d9c-498c-8b55-932214421e52 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:48.624932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:48.674563 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d963e9c8-9d9c-498c-8b55-932214421e52 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:48.714571 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566b4626-6a5c-4e2d-9685-1ff9be73bb29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:48.715221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566b4626-6a5c-4e2d-9685-1ff9be73bb29 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:48.715612 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566b4626-6a5c-4e2d-9685-1ff9be73bb29 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:48.715877 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566b4626-6a5c-4e2d-9685-1ff9be73bb29 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:48.716555 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566b4626-6a5c-4e2d-9685-1ff9be73bb29 None admin] RBAC: Adding query filter params (name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:48.716886 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566b4626-6a5c-4e2d-9685-1ff9be73bb29 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:48.718318 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566b4626-6a5c-4e2d-9685-1ff9be73bb29 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:48.732952 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 457/927] 10.222.0.36 () {56 vars in 1103 bytes} [Mon Oct 31 14:46:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:46:48.734416 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57932277-bafb-4ddf-a977-d1e357067e84 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:48.853511 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57932277-bafb-4ddf-a977-d1e357067e84 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:48.894765 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61300ddb-5445-4030-8bc5-bf0b7a883e25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:48.895404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61300ddb-5445-4030-8bc5-bf0b7a883e25 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:48.895749 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61300ddb-5445-4030-8bc5-bf0b7a883e25 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:48.896060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61300ddb-5445-4030-8bc5-bf0b7a883e25 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:48.906938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61300ddb-5445-4030-8bc5-bf0b7a883e25 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:48.909267 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61300ddb-5445-4030-8bc5-bf0b7a883e25 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:48.925665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-61300ddb-5445-4030-8bc5-bf0b7a883e25 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:48.925665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:48.925665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:48.925665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:48.925665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:48.925665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:48.925665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:48.925665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:48.925665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:48.925665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:48.925665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:48.925665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:48.925665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:46:48.925665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:46:48.925665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:46:48.925665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:46:48.926688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:48.926688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:48.926688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:48.926688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:48.926688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:48.926688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:48.926688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:48.926688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:48.926688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:48.926688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:48.926688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:48.926688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:48.926688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:48.926688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:48.926688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:48.926688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:48.926688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:46:48.928124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:46:48.928124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:48.928124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:46:48.928124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 471/928] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:46:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:46:48.932965 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566b4626-6a5c-4e2d-9685-1ff9be73bb29 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:49.042368 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566b4626-6a5c-4e2d-9685-1ff9be73bb29 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:49.059870 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:49.081227 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5841e6-388a-45b5-9e0e-1ad7d42b9fc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:49.081918 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5841e6-388a-45b5-9e0e-1ad7d42b9fc2 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:49.082216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5841e6-388a-45b5-9e0e-1ad7d42b9fc2 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:49.082555 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5841e6-388a-45b5-9e0e-1ad7d42b9fc2 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:49.083230 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5841e6-388a-45b5-9e0e-1ad7d42b9fc2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:49.083533 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5841e6-388a-45b5-9e0e-1ad7d42b9fc2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:49.084939 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5841e6-388a-45b5-9e0e-1ad7d42b9fc2 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:49.098261 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 458/929] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:46:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:46:49.104455 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61300ddb-5445-4030-8bc5-bf0b7a883e25 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:49.159157 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:49.214994 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61300ddb-5445-4030-8bc5-bf0b7a883e25 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:49.258500 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713c57c5-4a17-42d8-967e-11efceda6c75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:49.259129 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713c57c5-4a17-42d8-967e-11efceda6c75 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:49.259409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713c57c5-4a17-42d8-967e-11efceda6c75 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:49.259723 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713c57c5-4a17-42d8-967e-11efceda6c75 None admin] PATH_INFO: `/v3/users/placement` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:49.268439 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713c57c5-4a17-42d8-967e-11efceda6c75 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:49.269926 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713c57c5-4a17-42d8-967e-11efceda6c75 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:49.285623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-713c57c5-4a17-42d8-967e-11efceda6c75 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 31 14:46:49.285623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:49.285623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:49.285623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:49.285623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:49.285623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:49.285623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:49.285623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:49.285623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:49.285623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:49.285623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:49.285623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:49.285623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:46:49.285623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:46:49.285623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:46:49.285623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:49.286577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:49.287489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:49.287489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:49.287489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:46:49.287489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:46:49.287489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:46:49.287489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:46:49.287489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:46:49.287489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:46:49.287489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 31 14:46:49.287489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:46:49.287489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 472/930] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:46:49 2022] GET /identity/v3/users/placement => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:46:49.293541 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5841e6-388a-45b5-9e0e-1ad7d42b9fc2 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:49.402555 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5841e6-388a-45b5-9e0e-1ad7d42b9fc2 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:49.440350 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c232db-8baf-4a1f-8c76-c8ac412f75be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:49.440963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c232db-8baf-4a1f-8c76-c8ac412f75be None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:49.441238 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c232db-8baf-4a1f-8c76-c8ac412f75be None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:49.441512 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c232db-8baf-4a1f-8c76-c8ac412f75be None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:49.442239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c232db-8baf-4a1f-8c76-c8ac412f75be None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:49.442546 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c232db-8baf-4a1f-8c76-c8ac412f75be None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:49.444000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c232db-8baf-4a1f-8c76-c8ac412f75be None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:49.463008 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 459/931] 10.222.0.36 () {56 vars in 1143 bytes} [Mon Oct 31 14:46:49 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:46:49.468682 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713c57c5-4a17-42d8-967e-11efceda6c75 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:49.583656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713c57c5-4a17-42d8-967e-11efceda6c75 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:49.594274 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:49.619305 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e272b0f7-93f2-4d82-8ad3-39775be5edf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:49.620042 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e272b0f7-93f2-4d82-8ad3-39775be5edf7 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:49.620380 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e272b0f7-93f2-4d82-8ad3-39775be5edf7 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:49.620680 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e272b0f7-93f2-4d82-8ad3-39775be5edf7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:49.697754 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e272b0f7-93f2-4d82-8ad3-39775be5edf7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:49.699212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e272b0f7-93f2-4d82-8ad3-39775be5edf7 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:49.819503 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 473/932] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:46:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 31 14:46:49.820457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c232db-8baf-4a1f-8c76-c8ac412f75be None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:49.859337 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:49.942367 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c232db-8baf-4a1f-8c76-c8ac412f75be None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:49.981730 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1268a9c6-e2c2-4bd8-9f32-6f6979409048 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:49.982452 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1268a9c6-e2c2-4bd8-9f32-6f6979409048 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:49.982841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1268a9c6-e2c2-4bd8-9f32-6f6979409048 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:49.983163 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1268a9c6-e2c2-4bd8-9f32-6f6979409048 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:49.994223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1268a9c6-e2c2-4bd8-9f32-6f6979409048 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:49.996098 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1268a9c6-e2c2-4bd8-9f32-6f6979409048 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:50.013630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-1268a9c6-e2c2-4bd8-9f32-6f6979409048 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:50.013630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:50.013630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:50.013630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:50.013630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:50.013630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:50.013630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:50.013630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:50.013630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:50.013630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:50.013630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:50.013630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:50.013630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:46:50.013630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:46:50.013630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:46:50.013630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:46:50.014734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:50.014734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:50.014734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:50.014734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:50.014734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:50.014734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:50.014734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:50.014734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:50.014734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:50.014734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:50.014734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:50.014734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:50.014734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:50.014734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:50.014734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:50.014734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:50.014734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:46:50.015649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:46:50.015649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:50.015649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:46:50.015649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 460/933] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:46:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:46:50.023110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e272b0f7-93f2-4d82-8ad3-39775be5edf7 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:50.035571 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:50.134303 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e272b0f7-93f2-4d82-8ad3-39775be5edf7 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:50.170588 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abc3795-33aa-4dd9-8cfb-2aa72e4618af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:50.171207 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abc3795-33aa-4dd9-8cfb-2aa72e4618af None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:50.171498 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abc3795-33aa-4dd9-8cfb-2aa72e4618af None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:50.171804 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abc3795-33aa-4dd9-8cfb-2aa72e4618af None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:50.172443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abc3795-33aa-4dd9-8cfb-2aa72e4618af None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:50.172802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abc3795-33aa-4dd9-8cfb-2aa72e4618af None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:50.174362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abc3795-33aa-4dd9-8cfb-2aa72e4618af None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:50.186655 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 474/934] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:46:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:46:50.193669 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1268a9c6-e2c2-4bd8-9f32-6f6979409048 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:50.291870 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:50.308883 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1268a9c6-e2c2-4bd8-9f32-6f6979409048 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:50.351653 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0377fa2-c6ba-4e52-a883-0ab797e311ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:50.352400 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0377fa2-c6ba-4e52-a883-0ab797e311ae None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:50.352680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0377fa2-c6ba-4e52-a883-0ab797e311ae None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:50.352953 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0377fa2-c6ba-4e52-a883-0ab797e311ae None admin] PATH_INFO: `/v3/projects/service` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:50.361458 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0377fa2-c6ba-4e52-a883-0ab797e311ae None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:50.363101 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0377fa2-c6ba-4e52-a883-0ab797e311ae None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:50.377186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-a0377fa2-c6ba-4e52-a883-0ab797e311ae None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:46:50.377186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:50.377186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:50.377186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:50.377186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:50.377186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:50.377186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:50.377186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:50.377186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:50.377186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:50.377186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:50.377186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:50.377186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:46:50.377186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:46:50.377186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:46:50.378091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:46:50.378091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:50.378091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:50.378091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:50.378091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:50.378091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:50.378091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:50.378091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:50.378091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:50.378091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:50.378091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:50.378091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:50.378091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:50.378091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:50.378091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:50.378091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:50.378091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:50.379087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:46:50.379087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:46:50.379087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:46:50.379087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:46:50.379087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:46:50.379087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:46:50.379087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:46:50.379087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:46:50.379087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 461/935] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:46:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:46:50.384686 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abc3795-33aa-4dd9-8cfb-2aa72e4618af None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:50.465055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:50.499717 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abc3795-33aa-4dd9-8cfb-2aa72e4618af None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:50.542856 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4da8de2-9bd4-4666-92be-a775296e70ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:50.543695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4da8de2-9bd4-4666-92be-a775296e70ee None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:50.543980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4da8de2-9bd4-4666-92be-a775296e70ee None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:50.544255 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4da8de2-9bd4-4666-92be-a775296e70ee None admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:50.544948 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4da8de2-9bd4-4666-92be-a775296e70ee None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:50.545245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4da8de2-9bd4-4666-92be-a775296e70ee None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:50.546923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4da8de2-9bd4-4666-92be-a775296e70ee None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:50.560625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 475/936] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:46:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:46:50.568293 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0377fa2-c6ba-4e52-a883-0ab797e311ae None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:50.678778 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0377fa2-c6ba-4e52-a883-0ab797e311ae None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:50.714038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84eb1e53-8fb9-4d20-a908-fc33a9ee8b30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:50.714663 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84eb1e53-8fb9-4d20-a908-fc33a9ee8b30 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:50.714942 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84eb1e53-8fb9-4d20-a908-fc33a9ee8b30 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:50.715213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84eb1e53-8fb9-4d20-a908-fc33a9ee8b30 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:50.715951 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84eb1e53-8fb9-4d20-a908-fc33a9ee8b30 None admin] RBAC: Adding query filter params (role.id=88d62c66100a4d04a45e3607135bdd0c, scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8, user.id=70ea2aaf529b46dabb70b71f4dcc3860) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:50.716256 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84eb1e53-8fb9-4d20-a908-fc33a9ee8b30 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:50.717679 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84eb1e53-8fb9-4d20-a908-fc33a9ee8b30 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:50.723841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 462/937] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:46:50 2022] GET /identity/v3/role_assignments?user.id=70ea2aaf529b46dabb70b71f4dcc3860&scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8&role.id=88d62c66100a4d04a45e3607135bdd0c => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:46:53.162344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c67e9c-a139-450b-be4c-b213b36736be None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:53.162910 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c67e9c-a139-450b-be4c-b213b36736be None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:53.162910 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c67e9c-a139-450b-be4c-b213b36736be None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:53.164243 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 476/938] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:46:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:46:53.169746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6420354b-527b-43ce-8c42-6b70d77deff3 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:53.170351 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6420354b-527b-43ce-8c42-6b70d77deff3 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:53.170466 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6420354b-527b-43ce-8c42-6b70d77deff3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:53.191196 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-6420354b-527b-43ce-8c42-6b70d77deff3 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:46:53.192639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6420354b-527b-43ce-8c42-6b70d77deff3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:46:53.197211 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:53.275611 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 463/939] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:46:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:46:53.549880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c67e9c-a139-450b-be4c-b213b36736be None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:53.660931 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c67e9c-a139-450b-be4c-b213b36736be None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:53.695169 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:53.701848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577388b2-cabc-4a23-83a4-df04406f816b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:53.702477 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577388b2-cabc-4a23-83a4-df04406f816b None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:53.702750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577388b2-cabc-4a23-83a4-df04406f816b None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:53.703014 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577388b2-cabc-4a23-83a4-df04406f816b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:53.714239 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577388b2-cabc-4a23-83a4-df04406f816b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:46:53.720212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-577388b2-cabc-4a23-83a4-df04406f816b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:46:53.720212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:53.720212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:53.720212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:53.720212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:53.720212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:53.720212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:53.720212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:53.720212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:53.720212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:53.720212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:53.720212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:53.720212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:46:53.720212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:46:53.720212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:46:53.720212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:46:53.721361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:46:53.721361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:46:53.721361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:46:53.721361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:46:53.721361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:53.721361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:53.721361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:53.721361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:53.721361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:53.721361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:53.721361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:53.721361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:53.721361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:53.721361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:53.721361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:53.721361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:53.722196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:53.722196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:53.722196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:53.722196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:53.722196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:46:53.722196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:46:53.722196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:46:53.722196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:46:53.722196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:46:53.722196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:46:53.722196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:46:53.722196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:46:53.722196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 477/940] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:46:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:46:53.727744 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6420354b-527b-43ce-8c42-6b70d77deff3 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:53.839136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6420354b-527b-43ce-8c42-6b70d77deff3 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:53.876186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4748929-2ca9-4d74-8c89-d32505ba2e31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:53.876860 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4748929-2ca9-4d74-8c89-d32505ba2e31 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:53.877162 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4748929-2ca9-4d74-8c89-d32505ba2e31 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:53.877441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4748929-2ca9-4d74-8c89-d32505ba2e31 None admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:53.878107 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4748929-2ca9-4d74-8c89-d32505ba2e31 None admin] RBAC: Adding query filter params (name=admin) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:53.878409 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4748929-2ca9-4d74-8c89-d32505ba2e31 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:53.879791 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4748929-2ca9-4d74-8c89-d32505ba2e31 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:53.890257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 464/941] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:46:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:46:53.896742 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577388b2-cabc-4a23-83a4-df04406f816b None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:54.012482 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577388b2-cabc-4a23-83a4-df04406f816b None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:54.051352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412e6113-6b31-4399-9054-e5c730d5f19d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:54.052005 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412e6113-6b31-4399-9054-e5c730d5f19d None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:54.052280 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412e6113-6b31-4399-9054-e5c730d5f19d None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:54.052546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412e6113-6b31-4399-9054-e5c730d5f19d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:54.062381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412e6113-6b31-4399-9054-e5c730d5f19d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:54.063877 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412e6113-6b31-4399-9054-e5c730d5f19d None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:54.079236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-412e6113-6b31-4399-9054-e5c730d5f19d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:54.079236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:54.079236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:54.079236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:54.079236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:54.079236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:54.079236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:54.079236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:54.079236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:54.079236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:54.079236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:54.079236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:54.079236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:46:54.079236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:46:54.079236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:46:54.079236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:46:54.080585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:54.080585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:54.080585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:54.080585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:54.080585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:54.080585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:54.080585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:54.080585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:54.080585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:54.080585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:54.080585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:54.080585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:54.080585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:54.080585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:54.080585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:54.080585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:54.080585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:46:54.081461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:46:54.081461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:54.081461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:46:54.081461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 478/942] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:46:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:46:54.086579 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4748929-2ca9-4d74-8c89-d32505ba2e31 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:54.090161 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:54.200718 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4748929-2ca9-4d74-8c89-d32505ba2e31 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:54.242807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfcbcb6-f180-4fd4-aa22-b61329162580 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:54.243541 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfcbcb6-f180-4fd4-aa22-b61329162580 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:54.243819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfcbcb6-f180-4fd4-aa22-b61329162580 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:54.244141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfcbcb6-f180-4fd4-aa22-b61329162580 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:54.244746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfcbcb6-f180-4fd4-aa22-b61329162580 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:54.245060 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfcbcb6-f180-4fd4-aa22-b61329162580 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:54.246432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfcbcb6-f180-4fd4-aa22-b61329162580 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:54.260461 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 465/943] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:46:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:46:54.266582 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412e6113-6b31-4399-9054-e5c730d5f19d None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:54.312165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:54.379214 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412e6113-6b31-4399-9054-e5c730d5f19d None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:54.418870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633845ee-c9be-4553-87e8-18c1b3ab6bba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:54.419528 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633845ee-c9be-4553-87e8-18c1b3ab6bba None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:54.419890 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633845ee-c9be-4553-87e8-18c1b3ab6bba None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:54.421212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633845ee-c9be-4553-87e8-18c1b3ab6bba None admin] PATH_INFO: `/v3/users/placement` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:54.429859 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633845ee-c9be-4553-87e8-18c1b3ab6bba None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:54.431373 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633845ee-c9be-4553-87e8-18c1b3ab6bba None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:54.444146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-633845ee-c9be-4553-87e8-18c1b3ab6bba None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 31 14:46:54.444146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:54.444146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:54.444146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:54.444146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:54.444146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:54.444146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:54.444146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:54.444146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:54.444146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:54.444146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:54.444146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:54.444146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:46:54.444146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:46:54.444146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:46:54.444146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:54.451201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:54.452704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:54.452704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:54.452704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:46:54.452704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:46:54.452704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:46:54.452704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:46:54.452704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:46:54.452704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:46:54.452704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 31 14:46:54.452704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:46:54.452704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 479/944] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:46:54 2022] GET /identity/v3/users/placement => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:46:54.454131 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfcbcb6-f180-4fd4-aa22-b61329162580 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:54.535211 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:54.580733 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfcbcb6-f180-4fd4-aa22-b61329162580 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:54.617719 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9cdaae-2764-446f-b19b-c60bae9ce84d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:54.618363 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9cdaae-2764-446f-b19b-c60bae9ce84d None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:54.618647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9cdaae-2764-446f-b19b-c60bae9ce84d None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:54.618922 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9cdaae-2764-446f-b19b-c60bae9ce84d None admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:54.619727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9cdaae-2764-446f-b19b-c60bae9ce84d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:54.620107 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9cdaae-2764-446f-b19b-c60bae9ce84d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:54.621546 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9cdaae-2764-446f-b19b-c60bae9ce84d None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:54.643942 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 466/945] 10.222.0.36 () {56 vars in 1143 bytes} [Mon Oct 31 14:46:54 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:46:54.647234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633845ee-c9be-4553-87e8-18c1b3ab6bba None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:54.759793 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633845ee-c9be-4553-87e8-18c1b3ab6bba None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:54.761332 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:54.798955 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26656c4-57d6-43b9-9b26-7dae2686a6a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:54.799606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26656c4-57d6-43b9-9b26-7dae2686a6a1 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:54.799886 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26656c4-57d6-43b9-9b26-7dae2686a6a1 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:54.800155 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26656c4-57d6-43b9-9b26-7dae2686a6a1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:54.810863 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26656c4-57d6-43b9-9b26-7dae2686a6a1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:54.812331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26656c4-57d6-43b9-9b26-7dae2686a6a1 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:54.829542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-e26656c4-57d6-43b9-9b26-7dae2686a6a1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:54.829542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:54.829542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:54.829542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:54.829542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:54.829542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:54.829542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:54.829542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:54.829542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:54.829542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:54.829542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:54.829542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:54.829542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:46:54.829542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:46:54.829542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:46:54.829542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:46:54.830527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:54.830527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:54.830527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:54.830527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:54.830527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:54.830527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:54.830527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:54.830527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:54.830527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:54.830527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:54.830527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:54.830527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:54.830527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:54.830527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:54.830527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:54.830527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:54.830527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:46:54.831529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:46:54.831529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:54.831529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:46:54.831529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 480/946] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:46:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:46:54.836586 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9cdaae-2764-446f-b19b-c60bae9ce84d None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:54.953968 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9cdaae-2764-446f-b19b-c60bae9ce84d None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:54.977448 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:54.993622 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdab99e9-85f4-493a-8a70-e1b7779dc271 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:54.994278 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdab99e9-85f4-493a-8a70-e1b7779dc271 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:54.994558 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdab99e9-85f4-493a-8a70-e1b7779dc271 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:54.994838 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdab99e9-85f4-493a-8a70-e1b7779dc271 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:54.995467 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdab99e9-85f4-493a-8a70-e1b7779dc271 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:54.995820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdab99e9-85f4-493a-8a70-e1b7779dc271 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:54.997154 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdab99e9-85f4-493a-8a70-e1b7779dc271 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:55.009751 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 467/947] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:46:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:46:55.015627 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26656c4-57d6-43b9-9b26-7dae2686a6a1 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:55.124791 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26656c4-57d6-43b9-9b26-7dae2686a6a1 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:55.163251 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e1ba0e-1d6e-4922-86fa-de8ce6caacde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:55.163909 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e1ba0e-1d6e-4922-86fa-de8ce6caacde None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:55.164184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e1ba0e-1d6e-4922-86fa-de8ce6caacde None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:55.164498 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e1ba0e-1d6e-4922-86fa-de8ce6caacde None admin] PATH_INFO: `/v3/projects/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:55.171802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e1ba0e-1d6e-4922-86fa-de8ce6caacde None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:55.173859 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e1ba0e-1d6e-4922-86fa-de8ce6caacde None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:55.186328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-34e1ba0e-1d6e-4922-86fa-de8ce6caacde None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:46:55.186328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:55.186328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:55.186328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:55.186328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:55.186328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:55.186328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:55.186328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:55.186328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:55.186328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:55.186328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:55.186328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:55.186328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:46:55.186328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:46:55.186328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:46:55.187400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:46:55.187400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:55.187400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:55.187400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:55.187400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:55.187400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:55.187400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:55.187400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:55.187400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:55.187400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:55.187400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:55.187400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:55.187400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:55.187400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:55.187400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:55.187400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:55.187400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:55.189182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:46:55.189182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:46:55.189182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:46:55.189182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:46:55.189182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:46:55.189182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:46:55.189182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:46:55.189182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:46:55.189182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 481/948] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:46:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 31 14:46:55.194621 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdab99e9-85f4-493a-8a70-e1b7779dc271 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:55.305350 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdab99e9-85f4-493a-8a70-e1b7779dc271 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:55.342502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5283f20-fd9a-45b0-a3a3-206abed88806 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:55.343152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5283f20-fd9a-45b0-a3a3-206abed88806 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:55.343497 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5283f20-fd9a-45b0-a3a3-206abed88806 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:55.344545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5283f20-fd9a-45b0-a3a3-206abed88806 None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:55.345452 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5283f20-fd9a-45b0-a3a3-206abed88806 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:55.345761 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5283f20-fd9a-45b0-a3a3-206abed88806 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:55.347335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5283f20-fd9a-45b0-a3a3-206abed88806 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:55.360359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 468/949] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:46:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:46:55.366828 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e1ba0e-1d6e-4922-86fa-de8ce6caacde None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:55.384650 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:55.475479 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e1ba0e-1d6e-4922-86fa-de8ce6caacde None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:55.511232 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f254cc0-6077-4aa5-be0e-2fda7d004220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:55.511923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f254cc0-6077-4aa5-be0e-2fda7d004220 None admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:55.512256 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f254cc0-6077-4aa5-be0e-2fda7d004220 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:55.512519 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f254cc0-6077-4aa5-be0e-2fda7d004220 None admin] PATH_INFO: `/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/70ea2aaf529b46dabb70b71f4dcc3860/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:55.514642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f254cc0-6077-4aa5-be0e-2fda7d004220 None admin] RBAC: Authorizing `identity:create_grant(project_id=4118e33b099f4cf0a3e3a46b211ba1d8, user_id=70ea2aaf529b46dabb70b71f4dcc3860, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:55.516868 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f254cc0-6077-4aa5-be0e-2fda7d004220 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:55.530935 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 482/950] 10.222.0.36 () {58 vars in 1439 bytes} [Mon Oct 31 14:46:55 2022] PUT /identity/v3/projects/4118e33b099f4cf0a3e3a46b211ba1d8/users/70ea2aaf529b46dabb70b71f4dcc3860/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:46:58.080164 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755a60c8-c120-44bb-ba93-c465d79a9063 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:58.080749 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755a60c8-c120-44bb-ba93-c465d79a9063 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:58.080749 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755a60c8-c120-44bb-ba93-c465d79a9063 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:58.082140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 469/951] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:46:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:46:58.087829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcaf124-d939-4ffe-9a99-1ef08b871172 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:58.088256 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcaf124-d939-4ffe-9a99-1ef08b871172 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:58.088393 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcaf124-d939-4ffe-9a99-1ef08b871172 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:58.106849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-7fcaf124-d939-4ffe-9a99-1ef08b871172 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:46:58.108090 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7fcaf124-d939-4ffe-9a99-1ef08b871172 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:46:58.177814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 483/952] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:46:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:46:58.481651 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755a60c8-c120-44bb-ba93-c465d79a9063 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:58.532823 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:58.600161 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755a60c8-c120-44bb-ba93-c465d79a9063 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:58.635202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5725c1b-c1b4-4eb1-8e8e-16ab9123b297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:58.636360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5725c1b-c1b4-4eb1-8e8e-16ab9123b297 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:58.636360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5725c1b-c1b4-4eb1-8e8e-16ab9123b297 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:58.636360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5725c1b-c1b4-4eb1-8e8e-16ab9123b297 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:58.647437 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5725c1b-c1b4-4eb1-8e8e-16ab9123b297 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 31 14:46:58.652240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-f5725c1b-c1b4-4eb1-8e8e-16ab9123b297 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:46:58.652240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:58.652240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:58.652240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:58.652240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:58.652240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:58.652240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:58.652240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:58.652240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:58.652240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:58.652240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:58.652240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:58.652240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 31 14:46:58.652240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 31 14:46:58.652240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 31 14:46:58.652240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 31 14:46:58.653639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 31 14:46:58.653639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 31 14:46:58.653639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 31 14:46:58.653639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 31 14:46:58.653639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:58.653639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:58.653639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:58.653639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:58.653639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:58.653639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:58.653639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:58.653639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:58.653639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:58.653639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:58.653639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:58.653639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:58.654531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:58.654531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:58.654531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:58.654531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:58.654531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 31 14:46:58.654531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 31 14:46:58.654531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 31 14:46:58.654531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 31 14:46:58.654531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 31 14:46:58.654531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 31 14:46:58.654531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 31 14:46:58.654531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:46:58.654531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 470/953] 10.222.0.36 () {56 vars in 1096 bytes} [Mon Oct 31 14:46:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:46:58.659657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcaf124-d939-4ffe-9a99-1ef08b871172 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:58.662962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:58.767520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcaf124-d939-4ffe-9a99-1ef08b871172 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:58.802746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01125995-36e3-4f52-a7ce-c775efecb42d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:58.803365 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01125995-36e3-4f52-a7ce-c775efecb42d None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:58.803674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01125995-36e3-4f52-a7ce-c775efecb42d None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:58.803988 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01125995-36e3-4f52-a7ce-c775efecb42d None admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:58.804669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01125995-36e3-4f52-a7ce-c775efecb42d None admin] RBAC: Adding query filter params (name=admin) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:58.804969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01125995-36e3-4f52-a7ce-c775efecb42d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:58.806366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01125995-36e3-4f52-a7ce-c775efecb42d None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:58.815484 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 484/954] 10.222.0.36 () {56 vars in 1099 bytes} [Mon Oct 31 14:46:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:46:58.821431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5725c1b-c1b4-4eb1-8e8e-16ab9123b297 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:58.924913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:58.928478 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5725c1b-c1b4-4eb1-8e8e-16ab9123b297 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:58.966762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3deaf038-3049-49fa-ace8-8d5520206ebe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:58.967396 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3deaf038-3049-49fa-ace8-8d5520206ebe None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:58.967707 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3deaf038-3049-49fa-ace8-8d5520206ebe None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:58.967974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3deaf038-3049-49fa-ace8-8d5520206ebe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:58.979513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3deaf038-3049-49fa-ace8-8d5520206ebe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:58.981006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3deaf038-3049-49fa-ace8-8d5520206ebe None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:58.997373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-3deaf038-3049-49fa-ace8-8d5520206ebe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:58.997373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:58.997373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:58.997373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:58.997373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:58.997373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:58.997373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:58.997373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:58.997373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:58.997373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:58.997373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:58.997373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:58.997373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:46:58.997373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:46:58.997373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:46:58.997373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:46:59.003803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:59.003803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:59.003803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:59.003803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:59.003803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:59.003803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:59.003803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:59.003803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:59.003803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:59.003803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:59.003803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:59.003803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:59.003803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:59.003803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:59.003803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:59.003803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:59.003803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:46:59.004989 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:46:59.004989 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:59.004989 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:46:59.004989 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 471/955] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:46:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:46:59.005289 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01125995-36e3-4f52-a7ce-c775efecb42d None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:59.073326 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:59.110068 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01125995-36e3-4f52-a7ce-c775efecb42d None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:59.143331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952bea89-3139-4fd4-8eea-b0d2b3b5d06e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:59.143966 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952bea89-3139-4fd4-8eea-b0d2b3b5d06e None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:59.144243 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952bea89-3139-4fd4-8eea-b0d2b3b5d06e None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:59.144515 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952bea89-3139-4fd4-8eea-b0d2b3b5d06e None admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:59.145156 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952bea89-3139-4fd4-8eea-b0d2b3b5d06e None admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:59.145459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952bea89-3139-4fd4-8eea-b0d2b3b5d06e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:59.146825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952bea89-3139-4fd4-8eea-b0d2b3b5d06e None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:59.159069 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 485/956] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:46:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:46:59.164889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3deaf038-3049-49fa-ace8-8d5520206ebe None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:59.276354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3deaf038-3049-49fa-ace8-8d5520206ebe None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:59.310151 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4120023-fc13-4667-a842-2adcc1afb001 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:59.310779 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4120023-fc13-4667-a842-2adcc1afb001 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:59.311051 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4120023-fc13-4667-a842-2adcc1afb001 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:59.311317 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4120023-fc13-4667-a842-2adcc1afb001 None admin] PATH_INFO: `/v3/users/placement` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:59.319637 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4120023-fc13-4667-a842-2adcc1afb001 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:59.321090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4120023-fc13-4667-a842-2adcc1afb001 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:59.332722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-b4120023-fc13-4667-a842-2adcc1afb001 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 31 14:46:59.332722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:59.332722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:59.332722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:59.332722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:59.332722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:59.332722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:59.332722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:59.332722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:59.332722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:59.332722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:59.332722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:59.332722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 31 14:46:59.332722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 31 14:46:59.332722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 31 14:46:59.332722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:59.333726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:59.334745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:59.334745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:59.334745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 31 14:46:59.334745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 31 14:46:59.334745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 31 14:46:59.334745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 31 14:46:59.334745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 31 14:46:59.334745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 31 14:46:59.334745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 31 14:46:59.334745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:46:59.334745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 472/957] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:46:59 2022] GET /identity/v3/users/placement => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:46:59.339902 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952bea89-3139-4fd4-8eea-b0d2b3b5d06e None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:59.435280 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952bea89-3139-4fd4-8eea-b0d2b3b5d06e None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:59.460702 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:59.469810 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07796c2e-f565-4a87-9707-3ff0b136927d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:59.470405 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07796c2e-f565-4a87-9707-3ff0b136927d None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:59.470684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07796c2e-f565-4a87-9707-3ff0b136927d None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:59.470954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07796c2e-f565-4a87-9707-3ff0b136927d None admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:59.471695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07796c2e-f565-4a87-9707-3ff0b136927d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:46:59.471998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07796c2e-f565-4a87-9707-3ff0b136927d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:59.473402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07796c2e-f565-4a87-9707-3ff0b136927d None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:59.490115 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 486/958] 10.222.0.36 () {56 vars in 1143 bytes} [Mon Oct 31 14:46:59 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:46:59.495846 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4120023-fc13-4667-a842-2adcc1afb001 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:59.511263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:59.597858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4120023-fc13-4667-a842-2adcc1afb001 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:59.630870 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc798c6-140d-47a7-a502-e3f23bdeb9fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:59.631457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc798c6-140d-47a7-a502-e3f23bdeb9fa None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:59.631758 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc798c6-140d-47a7-a502-e3f23bdeb9fa None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:59.632029 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc798c6-140d-47a7-a502-e3f23bdeb9fa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:59.696273 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc798c6-140d-47a7-a502-e3f23bdeb9fa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:59.697699 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc798c6-140d-47a7-a502-e3f23bdeb9fa None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:59.710688 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:46:59.793267 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 473/959] 10.222.0.36 () {58 vars in 1282 bytes} [Mon Oct 31 14:46:59 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 31 14:46:59.799193 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07796c2e-f565-4a87-9707-3ff0b136927d None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:46:59.895136 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07796c2e-f565-4a87-9707-3ff0b136927d None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:46:59.928045 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a59729e-60c2-4814-bbdc-135970d264a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:46:59.928647 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a59729e-60c2-4814-bbdc-135970d264a8 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:46:59.928921 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a59729e-60c2-4814-bbdc-135970d264a8 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:46:59.929184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a59729e-60c2-4814-bbdc-135970d264a8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:46:59.939288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a59729e-60c2-4814-bbdc-135970d264a8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:46:59.940733 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a59729e-60c2-4814-bbdc-135970d264a8 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:46:59.954767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-8a59729e-60c2-4814-bbdc-135970d264a8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:59.954767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:46:59.954767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:46:59.954767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:46:59.954767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:46:59.954767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:46:59.954767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:46:59.954767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:46:59.954767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:46:59.954767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:46:59.954767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:46:59.954767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:46:59.954767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 31 14:46:59.954767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 31 14:46:59.954767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 31 14:46:59.954767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 31 14:46:59.955880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:46:59.955880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:46:59.955880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:46:59.955880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:46:59.955880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:46:59.955880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:46:59.955880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:46:59.955880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:46:59.955880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:46:59.955880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:46:59.955880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:46:59.955880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:46:59.955880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:46:59.955880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:46:59.955880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:46:59.955880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:46:59.955880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 31 14:46:59.956807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 31 14:46:59.956807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 31 14:46:59.956807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:46:59.956807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 487/960] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:46:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:46:59.961410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc798c6-140d-47a7-a502-e3f23bdeb9fa None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:47:00.057138 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc798c6-140d-47a7-a502-e3f23bdeb9fa None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:47:00.070142 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:47:00.091844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80c9b73-10b9-4a47-8ced-637b908a2e26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:47:00.092431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80c9b73-10b9-4a47-8ced-637b908a2e26 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:00.092708 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80c9b73-10b9-4a47-8ced-637b908a2e26 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:00.092975 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80c9b73-10b9-4a47-8ced-637b908a2e26 None admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:00.093650 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80c9b73-10b9-4a47-8ced-637b908a2e26 None admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:47:00.094573 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80c9b73-10b9-4a47-8ced-637b908a2e26 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:47:00.095947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80c9b73-10b9-4a47-8ced-637b908a2e26 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:47:00.106854 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 474/961] 10.222.0.36 () {56 vars in 1109 bytes} [Mon Oct 31 14:46:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:47:00.112591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a59729e-60c2-4814-bbdc-135970d264a8 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:47:00.147256 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:47:00.216781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a59729e-60c2-4814-bbdc-135970d264a8 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:47:00.261949 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6a0921-21d3-4d97-ab3d-9b76c921551b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:47:00.262571 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6a0921-21d3-4d97-ab3d-9b76c921551b None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:00.262853 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6a0921-21d3-4d97-ab3d-9b76c921551b None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:00.263128 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6a0921-21d3-4d97-ab3d-9b76c921551b None admin] PATH_INFO: `/v3/projects/service` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:00.270295 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6a0921-21d3-4d97-ab3d-9b76c921551b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:47:00.271760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6a0921-21d3-4d97-ab3d-9b76c921551b None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:47:00.283302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application [None req-dd6a0921-21d3-4d97-ab3d-9b76c921551b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:47:00.283302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:47:00.283302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:47:00.283302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:47:00.283302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:47:00.283302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:47:00.283302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:47:00.283302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:47:00.283302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:47:00.283302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:47:00.283302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:47:00.283302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:47:00.283302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 31 14:47:00.283302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 31 14:47:00.283302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 31 14:47:00.284363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 31 14:47:00.284363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:47:00.284363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:47:00.284363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:47:00.284363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:47:00.284363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:47:00.284363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:47:00.284363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:47:00.284363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application with Lock( Oct 31 14:47:00.284363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:47:00.284363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:47:00.284363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:47:00.284363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:47:00.284363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:47:00.284363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:47:00.284363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:47:00.284363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:47:00.285453 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 31 14:47:00.285453 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 31 14:47:00.285453 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 31 14:47:00.285453 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 31 14:47:00.285453 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 31 14:47:00.285453 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 31 14:47:00.285453 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 31 14:47:00.285453 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: ERROR keystone.server.flask.application Oct 31 14:47:00.285453 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 488/962] 10.222.0.36 () {56 vars in 1111 bytes} [Mon Oct 31 14:47:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:47:00.290260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80c9b73-10b9-4a47-8ced-637b908a2e26 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:47:00.387323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80c9b73-10b9-4a47-8ced-637b908a2e26 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:47:00.420419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95acd7c3-ae54-423d-91f3-f6cb585b99bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:47:00.421004 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95acd7c3-ae54-423d-91f3-f6cb585b99bf None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:00.421282 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95acd7c3-ae54-423d-91f3-f6cb585b99bf None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:00.421553 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95acd7c3-ae54-423d-91f3-f6cb585b99bf None admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:00.422262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95acd7c3-ae54-423d-91f3-f6cb585b99bf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:47:00.422565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95acd7c3-ae54-423d-91f3-f6cb585b99bf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:47:00.424017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95acd7c3-ae54-423d-91f3-f6cb585b99bf None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:47:00.435415 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 475/963] 10.222.0.36 () {56 vars in 1148 bytes} [Mon Oct 31 14:47:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:47:00.441965 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6a0921-21d3-4d97-ab3d-9b76c921551b None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:47:00.534130 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:47:00.540504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6a0921-21d3-4d97-ab3d-9b76c921551b None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:47:00.573540 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5583fc3-cac5-447b-aef9-2f2db4dbdc8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:47:00.574136 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5583fc3-cac5-447b-aef9-2f2db4dbdc8d None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:00.574410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5583fc3-cac5-447b-aef9-2f2db4dbdc8d None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:00.574678 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5583fc3-cac5-447b-aef9-2f2db4dbdc8d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:00.575387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5583fc3-cac5-447b-aef9-2f2db4dbdc8d None admin] RBAC: Adding query filter params (role.id=57572a37675d4c418e13794a7b8f3a5d, scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8, user.id=70ea2aaf529b46dabb70b71f4dcc3860) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:47:00.575743 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5583fc3-cac5-447b-aef9-2f2db4dbdc8d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:47:00.577149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5583fc3-cac5-447b-aef9-2f2db4dbdc8d None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:47:00.583309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 489/964] 10.222.0.36 () {56 vars in 1374 bytes} [Mon Oct 31 14:47:00 2022] GET /identity/v3/role_assignments?user.id=70ea2aaf529b46dabb70b71f4dcc3860&scope.project.id=4118e33b099f4cf0a3e3a46b211ba1d8&role.id=57572a37675d4c418e13794a7b8f3a5d => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:47:02.727878 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd49379a-6fea-4ec6-9b59-770a276a2b91 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:02.728468 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd49379a-6fea-4ec6-9b59-770a276a2b91 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:02.728468 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd49379a-6fea-4ec6-9b59-770a276a2b91 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:02.729743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 476/965] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:47:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:47:02.734904 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4c627b-2009-4802-8735-56d08639ce2d None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:02.735298 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4c627b-2009-4802-8735-56d08639ce2d None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:02.735439 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4c627b-2009-4802-8735-56d08639ce2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:02.753610 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-fb4c627b-2009-4802-8735-56d08639ce2d None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:47:02.754773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-fb4c627b-2009-4802-8735-56d08639ce2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:47:02.819691 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 490/966] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:47:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:47:03.088036 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd49379a-6fea-4ec6-9b59-770a276a2b91 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:47:03.119005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:47:03.188979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd49379a-6fea-4ec6-9b59-770a276a2b91 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:47:03.225722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e85372-b306-4cd8-b8a6-04bd41214c5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:47:03.226754 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e85372-b306-4cd8-b8a6-04bd41214c5a None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:03.227214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e85372-b306-4cd8-b8a6-04bd41214c5a None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:03.227780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e85372-b306-4cd8-b8a6-04bd41214c5a None admin] PATH_INFO: `/v3/services/placement` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:03.228801 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e85372-b306-4cd8-b8a6-04bd41214c5a None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:47:03.231151 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e85372-b306-4cd8-b8a6-04bd41214c5a None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:47:03.241192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-a3e85372-b306-4cd8-b8a6-04bd41214c5a None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 31 14:47:03.241192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:47:03.241192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:47:03.241192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:47:03.241192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:47:03.241192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:47:03.241192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:47:03.241192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:47:03.241192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 31 14:47:03.241192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:47:03.241192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:47:03.241192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:47:03.241192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:47:03.241192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:47:03.241192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:47:03.241192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:47:03.241192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:47:03.242732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:47:03.242732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:47:03.242732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:47:03.242732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:47:03.242732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:47:03.242732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:47:03.242732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:47:03.242732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:47:03.242732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:47:03.242732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:47:03.242732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:47:03.242732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:47:03.242732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:47:03.242732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:47:03.242732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:47:03.242732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:47:03.244038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:47:03.244038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:47:03.244038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:47:03.244038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:47:03.244038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:47:03.244038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:47:03.244038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:47:03.244038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:47:03.244038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:47:03.244038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:47:03.244038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:47:03.244038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:47:03.244038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 31 14:47:03.244038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:47:03.244038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 477/967] 10.222.0.36 () {56 vars in 1117 bytes} [Mon Oct 31 14:47:03 2022] GET /identity/v3/services/placement => generated 90 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:47:03.248386 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4c627b-2009-4802-8735-56d08639ce2d None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:47:03.339515 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:47:03.349078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4c627b-2009-4802-8735-56d08639ce2d None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:47:03.382524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143e5951-d7fd-4960-bea2-b0c85410a394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:47:03.383118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143e5951-d7fd-4960-bea2-b0c85410a394 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:03.383395 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143e5951-d7fd-4960-bea2-b0c85410a394 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:03.383695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143e5951-d7fd-4960-bea2-b0c85410a394 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:03.384313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143e5951-d7fd-4960-bea2-b0c85410a394 None admin] RBAC: Adding query filter params (name=placement) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:47:03.384612 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143e5951-d7fd-4960-bea2-b0c85410a394 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:47:03.385971 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143e5951-d7fd-4960-bea2-b0c85410a394 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:47:03.390798 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 491/968] 10.222.0.36 () {56 vars in 1116 bytes} [Mon Oct 31 14:47:03 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:47:03.396606 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e85372-b306-4cd8-b8a6-04bd41214c5a None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:47:03.489263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:47:03.495134 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e85372-b306-4cd8-b8a6-04bd41214c5a None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:47:03.527512 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cd10a0-7be4-4554-b7fd-be8c0036cdf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:47:03.528146 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cd10a0-7be4-4554-b7fd-be8c0036cdf7 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:03.528426 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cd10a0-7be4-4554-b7fd-be8c0036cdf7 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:03.528699 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cd10a0-7be4-4554-b7fd-be8c0036cdf7 None admin] PATH_INFO: `/v3/services` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:03.529327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cd10a0-7be4-4554-b7fd-be8c0036cdf7 None admin] RBAC: Adding query filter params (type=placement) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:47:03.529627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cd10a0-7be4-4554-b7fd-be8c0036cdf7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:47:03.531513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cd10a0-7be4-4554-b7fd-be8c0036cdf7 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:47:03.536092 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 478/969] 10.222.0.36 () {56 vars in 1116 bytes} [Mon Oct 31 14:47:03 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:47:05.685264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d4f8db-169f-4d31-8e43-b1808d870f6c None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:05.685800 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d4f8db-169f-4d31-8e43-b1808d870f6c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:05.685800 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d4f8db-169f-4d31-8e43-b1808d870f6c None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:05.687728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 492/970] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:47:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:47:05.693041 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac54e4ae-5f54-403f-a2fd-2668b50c7f44 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:05.693490 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac54e4ae-5f54-403f-a2fd-2668b50c7f44 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:05.693592 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac54e4ae-5f54-403f-a2fd-2668b50c7f44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:05.711678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-ac54e4ae-5f54-403f-a2fd-2668b50c7f44 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:47:05.712852 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ac54e4ae-5f54-403f-a2fd-2668b50c7f44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:47:05.776871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 479/971] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:47:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:47:06.042944 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d4f8db-169f-4d31-8e43-b1808d870f6c None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:47:06.140529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d4f8db-169f-4d31-8e43-b1808d870f6c None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:47:06.173961 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf40e274-ee97-4758-be27-4fc1383335cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:47:06.174547 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf40e274-ee97-4758-be27-4fc1383335cb None admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:06.174824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf40e274-ee97-4758-be27-4fc1383335cb None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:06.175095 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf40e274-ee97-4758-be27-4fc1383335cb None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:06.175770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf40e274-ee97-4758-be27-4fc1383335cb None admin] RBAC: Authorizing `identity:create_service()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:47:06.177113 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf40e274-ee97-4758-be27-4fc1383335cb None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:47:06.190961 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 493/972] 10.222.0.36 () {60 vars in 1141 bytes} [Mon Oct 31 14:47:06 2022] POST /identity/v3/services => generated 246 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:47:08.342992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabd6c89-d94e-4500-8497-1235d640a222 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:08.343544 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabd6c89-d94e-4500-8497-1235d640a222 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:08.344196 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabd6c89-d94e-4500-8497-1235d640a222 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:08.345539 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 480/973] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:47:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:47:08.350752 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9440da-1802-46e9-aa7f-b81e9e6925ab None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:08.351174 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9440da-1802-46e9-aa7f-b81e9e6925ab None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:08.351271 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9440da-1802-46e9-aa7f-b81e9e6925ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:08.369059 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-ac9440da-1802-46e9-aa7f-b81e9e6925ab None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:47:08.370220 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ac9440da-1802-46e9-aa7f-b81e9e6925ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:47:08.379223 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:47:08.440901 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 494/974] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:47:08 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:47:08.716682 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabd6c89-d94e-4500-8497-1235d640a222 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:47:08.805529 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:47:08.817153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabd6c89-d94e-4500-8497-1235d640a222 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:47:08.857722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c87e3ea-770d-44f4-8f82-7045ef734428 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:47:08.858346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c87e3ea-770d-44f4-8f82-7045ef734428 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:08.859676 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c87e3ea-770d-44f4-8f82-7045ef734428 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:08.860041 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c87e3ea-770d-44f4-8f82-7045ef734428 None admin] PATH_INFO: `/v3/services/placement` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:08.861022 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c87e3ea-770d-44f4-8f82-7045ef734428 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:47:08.862920 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c87e3ea-770d-44f4-8f82-7045ef734428 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:47:08.871962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-5c87e3ea-770d-44f4-8f82-7045ef734428 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 31 14:47:08.871962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:47:08.871962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:47:08.871962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:47:08.871962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:47:08.871962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:47:08.871962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:47:08.871962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:47:08.871962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 31 14:47:08.871962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:47:08.871962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:47:08.871962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:47:08.871962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:47:08.871962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:47:08.871962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:47:08.871962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:47:08.871962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:47:08.873140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:47:08.873140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:47:08.873140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:47:08.873140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:47:08.873140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:47:08.873140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:47:08.873140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:47:08.873140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:47:08.873140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:47:08.873140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:47:08.873140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:47:08.873140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:47:08.873140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:47:08.873140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:47:08.873140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:47:08.873140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:47:08.874000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:47:08.874000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:47:08.874000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:47:08.874000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:47:08.874000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:47:08.874000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:47:08.874000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:47:08.874000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:47:08.874000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:47:08.874000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:47:08.874000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:47:08.874000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:47:08.874000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 31 14:47:08.874000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:47:08.874000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 481/975] 10.222.0.36 () {56 vars in 1117 bytes} [Mon Oct 31 14:47:08 2022] GET /identity/v3/services/placement => generated 90 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:47:08.879865 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9440da-1802-46e9-aa7f-b81e9e6925ab None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:47:08.989589 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9440da-1802-46e9-aa7f-b81e9e6925ab None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:47:09.024338 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98685d8-56fa-4029-a33e-09f5626647b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:47:09.024944 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98685d8-56fa-4029-a33e-09f5626647b0 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:09.025222 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98685d8-56fa-4029-a33e-09f5626647b0 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:09.025496 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98685d8-56fa-4029-a33e-09f5626647b0 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:09.026121 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98685d8-56fa-4029-a33e-09f5626647b0 None admin] RBAC: Adding query filter params (name=placement) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:47:09.026425 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98685d8-56fa-4029-a33e-09f5626647b0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:47:09.027812 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98685d8-56fa-4029-a33e-09f5626647b0 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:47:09.033058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 495/976] 10.222.0.36 () {56 vars in 1116 bytes} [Mon Oct 31 14:47:08 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:47:09.039341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c87e3ea-770d-44f4-8f82-7045ef734428 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:47:09.153304 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c87e3ea-770d-44f4-8f82-7045ef734428 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:47:09.195658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f884aa5-2d78-4b15-951b-0f7b16960aca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:47:09.196358 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f884aa5-2d78-4b15-951b-0f7b16960aca None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:09.196564 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f884aa5-2d78-4b15-951b-0f7b16960aca None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:09.196835 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f884aa5-2d78-4b15-951b-0f7b16960aca None admin] PATH_INFO: `/v3/endpoints` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:09.197550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f884aa5-2d78-4b15-951b-0f7b16960aca None admin] RBAC: Adding query filter params (interface=public, service_id=a8bb064b95ec40649bc075f3554b2f52, region_id=RegionOne) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:47:09.197858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f884aa5-2d78-4b15-951b-0f7b16960aca None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:47:09.199275 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f884aa5-2d78-4b15-951b-0f7b16960aca None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:47:09.205048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 482/977] 10.222.0.36 () {56 vars in 1251 bytes} [Mon Oct 31 14:47:09 2022] GET /identity/v3/endpoints?service_id=a8bb064b95ec40649bc075f3554b2f52&interface=public®ion_id=RegionOne => generated 194 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:47:09.210781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98685d8-56fa-4029-a33e-09f5626647b0 None admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:47:09.215405 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:47:09.326453 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98685d8-56fa-4029-a33e-09f5626647b0 None admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:47:09.369750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d6f4ae-74d5-4171-8aa7-4a49379025f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:47:09.370426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d6f4ae-74d5-4171-8aa7-4a49379025f8 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:09.371515 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d6f4ae-74d5-4171-8aa7-4a49379025f8 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:09.371838 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d6f4ae-74d5-4171-8aa7-4a49379025f8 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:09.372446 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d6f4ae-74d5-4171-8aa7-4a49379025f8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:47:09.373804 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d6f4ae-74d5-4171-8aa7-4a49379025f8 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:47:09.385005 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 496/978] 10.222.0.36 () {56 vars in 1087 bytes} [Mon Oct 31 14:47:09 2022] GET /identity/v3/services => generated 2191 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:47:11.533727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c0d7fe-23bb-4a51-af2a-e10d0a1e379a None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:11.534194 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c0d7fe-23bb-4a51-af2a-e10d0a1e379a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:11.534294 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c0d7fe-23bb-4a51-af2a-e10d0a1e379a None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:11.535647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 483/979] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:47:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:47:11.540922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1948522a-3b97-4025-b2af-627b5c8bbb1b None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:11.541325 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1948522a-3b97-4025-b2af-627b5c8bbb1b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:11.541470 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1948522a-3b97-4025-b2af-627b5c8bbb1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:11.559439 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-1948522a-3b97-4025-b2af-627b5c8bbb1b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:47:11.560656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1948522a-3b97-4025-b2af-627b5c8bbb1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:47:11.619828 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:47:11.625660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 497/980] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:47:11 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 31 14:47:11.894513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c0d7fe-23bb-4a51-af2a-e10d0a1e379a None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:47:11.913845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:47:11.993619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c0d7fe-23bb-4a51-af2a-e10d0a1e379a None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:47:12.026880 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd75b89a-dad5-4071-834f-77bace2965e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:47:12.027497 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd75b89a-dad5-4071-834f-77bace2965e1 None admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:12.027807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd75b89a-dad5-4071-834f-77bace2965e1 None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:12.028082 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd75b89a-dad5-4071-834f-77bace2965e1 None admin] PATH_INFO: `/v3/services/placement` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:12.028662 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd75b89a-dad5-4071-834f-77bace2965e1 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:47:12.030073 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd75b89a-dad5-4071-834f-77bace2965e1 None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:47:12.038858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application [None req-dd75b89a-dad5-4071-834f-77bace2965e1 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 31 14:47:12.038858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:47:12.038858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 31 14:47:12.038858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 31 14:47:12.038858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 31 14:47:12.038858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 31 14:47:12.038858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 31 14:47:12.038858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:47:12.038858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 31 14:47:12.038858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:47:12.038858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 31 14:47:12.038858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:47:12.038858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 31 14:47:12.038858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 31 14:47:12.038858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 31 14:47:12.038858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 31 14:47:12.038858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 31 14:47:12.040030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 31 14:47:12.040030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 31 14:47:12.040030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 31 14:47:12.040030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 31 14:47:12.040030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 31 14:47:12.040030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 31 14:47:12.040030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 31 14:47:12.040030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 31 14:47:12.040030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 31 14:47:12.040030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 31 14:47:12.040030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 31 14:47:12.040030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 31 14:47:12.040030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 31 14:47:12.040030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 31 14:47:12.040030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 31 14:47:12.040030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.get_or_create( Oct 31 14:47:12.040893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 31 14:47:12.040893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application with Lock( Oct 31 14:47:12.040893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 31 14:47:12.040893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self._enter() Oct 31 14:47:12.040893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 31 14:47:12.040893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 31 14:47:12.040893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 31 14:47:12.040893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application return self.creator() Oct 31 14:47:12.040893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 31 14:47:12.040893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application created_value = creator( Oct 31 14:47:12.040893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 31 14:47:12.040893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 31 14:47:12.040893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 31 14:47:12.040893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: ERROR keystone.server.flask.application Oct 31 14:47:12.040893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 484/981] 10.222.0.36 () {56 vars in 1117 bytes} [Mon Oct 31 14:47:11 2022] GET /identity/v3/services/placement => generated 90 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 31 14:47:12.046055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1948522a-3b97-4025-b2af-627b5c8bbb1b None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:47:12.141964 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1948522a-3b97-4025-b2af-627b5c8bbb1b None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:47:12.174703 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a66686-b4b4-4861-b0f7-33dd454cc8a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:47:12.175294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a66686-b4b4-4861-b0f7-33dd454cc8a0 None admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:12.175597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a66686-b4b4-4861-b0f7-33dd454cc8a0 None admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:12.175876 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a66686-b4b4-4861-b0f7-33dd454cc8a0 None admin] PATH_INFO: `/v3/services` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:12.176496 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a66686-b4b4-4861-b0f7-33dd454cc8a0 None admin] RBAC: Adding query filter params (name=placement) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:47:12.176800 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a66686-b4b4-4861-b0f7-33dd454cc8a0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:47:12.178158 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a66686-b4b4-4861-b0f7-33dd454cc8a0 None admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:47:12.183090 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 498/982] 10.222.0.36 () {56 vars in 1116 bytes} [Mon Oct 31 14:47:12 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:47:12.189481 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd75b89a-dad5-4071-834f-77bace2965e1 None admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:47:12.278249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:47:12.296780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd75b89a-dad5-4071-834f-77bace2965e1 None admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:47:12.329860 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77258a6e-1a1f-4e3a-8117-210c69f2174b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:47:12.330460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77258a6e-1a1f-4e3a-8117-210c69f2174b None admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:47:12.330742 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77258a6e-1a1f-4e3a-8117-210c69f2174b None admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:47:12.331014 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77258a6e-1a1f-4e3a-8117-210c69f2174b None admin] PATH_INFO: `/v3/endpoints` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:47:12.331703 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77258a6e-1a1f-4e3a-8117-210c69f2174b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:47:12.333071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77258a6e-1a1f-4e3a-8117-210c69f2174b None admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:47:12.352239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 485/983] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:47:12 2022] POST /identity/v3/endpoints => generated 331 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:47:22.183196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=69 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:47:22.348844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:18.627692 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4bf21c-f68b-422e-8a9d-997ad6d0bb2f None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:18.628261 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4bf21c-f68b-422e-8a9d-997ad6d0bb2f None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:18.628261 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4bf21c-f68b-422e-8a9d-997ad6d0bb2f None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:18.629990 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 499/984] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:48:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:48:18.635394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a31950c-c0f2-4001-9487-071c7b95dbf5 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:18.635922 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a31950c-c0f2-4001-9487-071c7b95dbf5 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:18.636030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a31950c-c0f2-4001-9487-071c7b95dbf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:18.718339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-3a31950c-c0f2-4001-9487-071c7b95dbf5 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:48:18.719580 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-3a31950c-c0f2-4001-9487-071c7b95dbf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:18.909105 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 486/985] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:48:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:48:19.242315 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f1e678-05f5-4c40-ad76-8ca0b17fd293 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:19.242619 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f1e678-05f5-4c40-ad76-8ca0b17fd293 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:19.242876 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f1e678-05f5-4c40-ad76-8ca0b17fd293 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:19.244381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 500/986] 10.222.0.36 () {54 vars in 965 bytes} [Mon Oct 31 14:48:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:48:19.250468 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7248c4-ecb6-44f9-8ee6-eaeb7d9b6dbc None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:19.250780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7248c4-ecb6-44f9-8ee6-eaeb7d9b6dbc None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:19.251088 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7248c4-ecb6-44f9-8ee6-eaeb7d9b6dbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:19.338078 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-4c7248c4-ecb6-44f9-8ee6-eaeb7d9b6dbc None None] MFA Rules not processed for user `14946d97fd384e2a8bff68c4f53c2cfe`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:48:19.339312 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-4c7248c4-ecb6-44f9-8ee6-eaeb7d9b6dbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:19.362737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:19.426295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 487/987] 10.222.0.36 () {58 vars in 1062 bytes} [Mon Oct 31 14:48:19 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:48:19.436054 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f1e678-05f5-4c40-ad76-8ca0b17fd293 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:48:19.437885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-20f1e678-05f5-4c40-ad76-8ca0b17fd293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:19.573833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f1e678-05f5-4c40-ad76-8ca0b17fd293 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:48:19.672575 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1432d5-d3f4-4f19-ad8a-58ff7c1bb26a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:48:19.673331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1432d5-d3f4-4f19-ad8a-58ff7c1bb26a service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:19.673603 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1432d5-d3f4-4f19-ad8a-58ff7c1bb26a service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:19.673895 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1432d5-d3f4-4f19-ad8a-58ff7c1bb26a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:19.675949 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-6d1432d5-d3f4-4f19-ad8a-58ff7c1bb26a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:19.743695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:19.813233 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1432d5-d3f4-4f19-ad8a-58ff7c1bb26a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:48:19.814783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1432d5-d3f4-4f19-ad8a-58ff7c1bb26a service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:48:19.837754 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 501/988] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:48:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:48:22.473208 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b8edb3-665c-4138-b43f-73ecc89fd4be None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:22.473763 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b8edb3-665c-4138-b43f-73ecc89fd4be None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:22.473763 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b8edb3-665c-4138-b43f-73ecc89fd4be None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:22.475271 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 488/989] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:48:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:48:22.481038 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99030b7-dfa2-4c04-b7c3-da0b95ca50de None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:22.481465 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99030b7-dfa2-4c04-b7c3-da0b95ca50de None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:22.481573 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99030b7-dfa2-4c04-b7c3-da0b95ca50de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:22.578023 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-f99030b7-dfa2-4c04-b7c3-da0b95ca50de None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:48:22.579588 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-f99030b7-dfa2-4c04-b7c3-da0b95ca50de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:22.692739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 502/990] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:48:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:48:22.983881 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b8edb3-665c-4138-b43f-73ecc89fd4be None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:48:22.985473 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-60b8edb3-665c-4138-b43f-73ecc89fd4be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:23.096882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b8edb3-665c-4138-b43f-73ecc89fd4be None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:48:23.175335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:23.178104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a1d25b-bf04-47ee-9af9-c2ec261510cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:48:23.178979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a1d25b-bf04-47ee-9af9-c2ec261510cb service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:23.179447 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a1d25b-bf04-47ee-9af9-c2ec261510cb service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:23.179918 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a1d25b-bf04-47ee-9af9-c2ec261510cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:23.181579 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a1a1d25b-bf04-47ee-9af9-c2ec261510cb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:23.295354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a1d25b-bf04-47ee-9af9-c2ec261510cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:48:23.296737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a1d25b-bf04-47ee-9af9-c2ec261510cb service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:48:23.315716 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 489/991] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:48:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:48:26.139217 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e6baf1-6b2a-4ce2-8117-d11f806bd01b None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:26.139831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e6baf1-6b2a-4ce2-8117-d11f806bd01b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:26.139831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e6baf1-6b2a-4ce2-8117-d11f806bd01b None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:26.141133 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 503/992] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:48:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:48:26.146532 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4925f95d-bd7f-419d-a9e9-bf6bc37e6cdb None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:26.146972 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4925f95d-bd7f-419d-a9e9-bf6bc37e6cdb None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:26.147108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4925f95d-bd7f-419d-a9e9-bf6bc37e6cdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:26.213405 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-4925f95d-bd7f-419d-a9e9-bf6bc37e6cdb None None] MFA Rules not processed for user `b0dd86f9993647219d6b755d3d7c09f5`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:48:26.214619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-4925f95d-bd7f-419d-a9e9-bf6bc37e6cdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:26.267659 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:26.313272 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 490/993] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:48:26 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:48:26.597826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e6baf1-6b2a-4ce2-8117-d11f806bd01b None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:48:26.599379 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-51e6baf1-6b2a-4ce2-8117-d11f806bd01b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:26.610258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:26.709722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e6baf1-6b2a-4ce2-8117-d11f806bd01b None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:48:26.784213 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555a21b2-758a-424e-8675-5a563cc37b97 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:48:26.784733 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555a21b2-758a-424e-8675-5a563cc37b97 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:26.785011 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555a21b2-758a-424e-8675-5a563cc37b97 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:26.785281 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555a21b2-758a-424e-8675-5a563cc37b97 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:26.786881 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-555a21b2-758a-424e-8675-5a563cc37b97 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:26.849426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:26.897873 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555a21b2-758a-424e-8675-5a563cc37b97 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:48:26.899364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555a21b2-758a-424e-8675-5a563cc37b97 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:48:26.919227 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 504/994] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:48:26 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:48:30.037215 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67975ba5-6b76-4b18-adf2-a70525cd62fc None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:30.037791 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67975ba5-6b76-4b18-adf2-a70525cd62fc None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:30.037791 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67975ba5-6b76-4b18-adf2-a70525cd62fc None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:30.039123 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 491/995] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:48:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:48:30.044632 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b940170-e873-4817-ac15-a65aa403c743 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:30.045114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b940170-e873-4817-ac15-a65aa403c743 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:30.045210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b940170-e873-4817-ac15-a65aa403c743 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:30.112104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-5b940170-e873-4817-ac15-a65aa403c743 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:48:30.113366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5b940170-e873-4817-ac15-a65aa403c743 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:30.213356 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 505/996] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:48:30 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:48:30.517241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca2f5b7-e783-4319-8be8-b0f626683262 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:30.517750 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca2f5b7-e783-4319-8be8-b0f626683262 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:30.518218 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca2f5b7-e783-4319-8be8-b0f626683262 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:30.520166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 492/997] 10.222.0.36 () {54 vars in 965 bytes} [Mon Oct 31 14:48:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:48:30.526444 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51f6e0b-f304-4337-839f-3d9383f3cf82 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:30.526734 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51f6e0b-f304-4337-839f-3d9383f3cf82 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:30.527044 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51f6e0b-f304-4337-839f-3d9383f3cf82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:30.549844 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:30.604299 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-d51f6e0b-f304-4337-839f-3d9383f3cf82 None None] MFA Rules not processed for user `14946d97fd384e2a8bff68c4f53c2cfe`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:48:30.606260 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d51f6e0b-f304-4337-839f-3d9383f3cf82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:30.681268 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 506/998] 10.222.0.36 () {58 vars in 1062 bytes} [Mon Oct 31 14:48:30 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:48:30.690986 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca2f5b7-e783-4319-8be8-b0f626683262 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:48:30.692545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-3ca2f5b7-e783-4319-8be8-b0f626683262 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:30.800115 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca2f5b7-e783-4319-8be8-b0f626683262 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:48:30.873858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d47dad-b5d5-408c-8bc9-f9ad824db245 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:48:30.874710 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d47dad-b5d5-408c-8bc9-f9ad824db245 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:30.874990 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d47dad-b5d5-408c-8bc9-f9ad824db245 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:30.875259 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d47dad-b5d5-408c-8bc9-f9ad824db245 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:30.876902 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-10d47dad-b5d5-408c-8bc9-f9ad824db245 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:30.883976 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:30.994063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d47dad-b5d5-408c-8bc9-f9ad824db245 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:48:30.996359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d47dad-b5d5-408c-8bc9-f9ad824db245 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:48:31.020997 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 493/999] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:48:30 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:48:34.520092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fea251-813b-4a2d-88ad-a636185dce0e None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:34.520736 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fea251-813b-4a2d-88ad-a636185dce0e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:34.520736 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fea251-813b-4a2d-88ad-a636185dce0e None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:34.522156 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 507/1000] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:48:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:48:34.529268 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9468162c-52a6-4845-addb-3a34d0b4ed68 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:34.530284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9468162c-52a6-4845-addb-3a34d0b4ed68 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:34.530284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9468162c-52a6-4845-addb-3a34d0b4ed68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:34.600976 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-9468162c-52a6-4845-addb-3a34d0b4ed68 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:48:34.602239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-9468162c-52a6-4845-addb-3a34d0b4ed68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:34.672576 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:34.719990 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 494/1001] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:48:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:48:35.005999 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fea251-813b-4a2d-88ad-a636185dce0e None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:48:35.007595 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-73fea251-813b-4a2d-88ad-a636185dce0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:35.117549 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fea251-813b-4a2d-88ad-a636185dce0e None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:48:35.130941 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:35.192991 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f212b998-9f07-45ef-bbce-9543b7176264 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:48:35.193604 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f212b998-9f07-45ef-bbce-9543b7176264 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:35.193880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f212b998-9f07-45ef-bbce-9543b7176264 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:35.194149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f212b998-9f07-45ef-bbce-9543b7176264 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:35.195800 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-f212b998-9f07-45ef-bbce-9543b7176264 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:35.313548 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f212b998-9f07-45ef-bbce-9543b7176264 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:48:35.314908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f212b998-9f07-45ef-bbce-9543b7176264 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:48:35.334189 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 508/1002] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:48:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:48:37.579536 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e898ea1-0c6d-49e0-b469-f3159bad3aa4 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:37.580160 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e898ea1-0c6d-49e0-b469-f3159bad3aa4 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:37.580290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e898ea1-0c6d-49e0-b469-f3159bad3aa4 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:37.581831 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 495/1003] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:48:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:48:37.588649 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d04f15-20e8-4b9b-af26-a7ccd9205534 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:37.589105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d04f15-20e8-4b9b-af26-a7ccd9205534 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:37.589324 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d04f15-20e8-4b9b-af26-a7ccd9205534 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:37.645987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:37.665249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-c0d04f15-20e8-4b9b-af26-a7ccd9205534 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:48:37.666479 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-c0d04f15-20e8-4b9b-af26-a7ccd9205534 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:37.772522 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 509/1004] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:48:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:48:38.054244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e898ea1-0c6d-49e0-b469-f3159bad3aa4 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:48:38.056161 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6e898ea1-0c6d-49e0-b469-f3159bad3aa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:38.180989 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e898ea1-0c6d-49e0-b469-f3159bad3aa4 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:48:38.270226 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6236a6cc-a677-4c73-aeb2-50a025ee3d20 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:48:38.270971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6236a6cc-a677-4c73-aeb2-50a025ee3d20 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:38.271289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6236a6cc-a677-4c73-aeb2-50a025ee3d20 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:38.271622 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6236a6cc-a677-4c73-aeb2-50a025ee3d20 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:38.273598 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6236a6cc-a677-4c73-aeb2-50a025ee3d20 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:38.294557 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:38.406194 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6236a6cc-a677-4c73-aeb2-50a025ee3d20 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:48:38.408060 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6236a6cc-a677-4c73-aeb2-50a025ee3d20 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:48:38.429453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 496/1005] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:48:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:48:40.716979 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b0f868-952c-4443-b1a8-34d832be7388 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:40.717429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b0f868-952c-4443-b1a8-34d832be7388 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:40.717507 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b0f868-952c-4443-b1a8-34d832be7388 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:40.719412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 510/1006] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:48:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:48:40.726021 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8fcaee-28ef-46db-bbee-dcb2d672567f None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:40.726483 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8fcaee-28ef-46db-bbee-dcb2d672567f None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:40.726810 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8fcaee-28ef-46db-bbee-dcb2d672567f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:40.799465 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-bc8fcaee-28ef-46db-bbee-dcb2d672567f None None] MFA Rules not processed for user `b0dd86f9993647219d6b755d3d7c09f5`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:48:40.800716 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-bc8fcaee-28ef-46db-bbee-dcb2d672567f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:40.880657 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:40.914804 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 497/1007] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:48:40 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:48:41.206878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b0f868-952c-4443-b1a8-34d832be7388 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:48:41.209269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b5b0f868-952c-4443-b1a8-34d832be7388 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:41.318504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b0f868-952c-4443-b1a8-34d832be7388 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:48:41.342089 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:41.400273 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892bea03-f864-44c1-b6b7-44be9c4386bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:48:41.400887 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892bea03-f864-44c1-b6b7-44be9c4386bb service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:41.401166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892bea03-f864-44c1-b6b7-44be9c4386bb service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:41.401449 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892bea03-f864-44c1-b6b7-44be9c4386bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:41.403742 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-892bea03-f864-44c1-b6b7-44be9c4386bb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:41.523321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892bea03-f864-44c1-b6b7-44be9c4386bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:48:41.524999 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892bea03-f864-44c1-b6b7-44be9c4386bb service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:48:41.548506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 511/1008] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:48:41 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:48:44.160809 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bb4382-b52d-4b2b-90f9-0968f5bf4b52 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:44.161379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bb4382-b52d-4b2b-90f9-0968f5bf4b52 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:44.161484 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bb4382-b52d-4b2b-90f9-0968f5bf4b52 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:44.162889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 498/1009] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:48:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:48:44.168905 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27102d82-b09e-4bbd-86e8-67478f988629 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:44.169343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27102d82-b09e-4bbd-86e8-67478f988629 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:44.169444 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27102d82-b09e-4bbd-86e8-67478f988629 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:44.234455 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:44.237364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-27102d82-b09e-4bbd-86e8-67478f988629 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:48:44.239059 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-27102d82-b09e-4bbd-86e8-67478f988629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:44.337664 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 512/1010] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:48:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:48:44.626324 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bb4382-b52d-4b2b-90f9-0968f5bf4b52 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:48:44.628025 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e5bb4382-b52d-4b2b-90f9-0968f5bf4b52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:44.740053 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bb4382-b52d-4b2b-90f9-0968f5bf4b52 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:48:44.819240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99986750-f4ac-402c-9fd2-f6d46db5adad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:48:44.819854 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99986750-f4ac-402c-9fd2-f6d46db5adad service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:44.820227 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99986750-f4ac-402c-9fd2-f6d46db5adad service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:44.820541 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99986750-f4ac-402c-9fd2-f6d46db5adad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:44.822327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-99986750-f4ac-402c-9fd2-f6d46db5adad service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:44.851586 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:44.941151 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99986750-f4ac-402c-9fd2-f6d46db5adad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:48:44.942586 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99986750-f4ac-402c-9fd2-f6d46db5adad service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:48:44.961498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 499/1011] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:48:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:48:47.219706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0450ebf2-e50e-4844-89e1-b3b6c6e363b1 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:47.220269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0450ebf2-e50e-4844-89e1-b3b6c6e363b1 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:47.220269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0450ebf2-e50e-4844-89e1-b3b6c6e363b1 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:47.221605 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 513/1012] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:48:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:48:47.226946 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fbf850-8fb2-4e48-8ffe-3f8f41856976 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:47.227458 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fbf850-8fb2-4e48-8ffe-3f8f41856976 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:47.227589 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fbf850-8fb2-4e48-8ffe-3f8f41856976 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:47.295712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-40fbf850-8fb2-4e48-8ffe-3f8f41856976 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:48:47.296939 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-40fbf850-8fb2-4e48-8ffe-3f8f41856976 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:47.373330 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:47.396113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 500/1013] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:48:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:48:47.684518 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0450ebf2-e50e-4844-89e1-b3b6c6e363b1 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:48:47.686068 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0450ebf2-e50e-4844-89e1-b3b6c6e363b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:47.790062 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0450ebf2-e50e-4844-89e1-b3b6c6e363b1 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:48:47.821236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:47.869781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c9ae52-fbfb-4447-b635-2a22705bc083 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:48:47.870418 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c9ae52-fbfb-4447-b635-2a22705bc083 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:47.870705 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c9ae52-fbfb-4447-b635-2a22705bc083 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:47.870993 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c9ae52-fbfb-4447-b635-2a22705bc083 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:47.873583 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-82c9ae52-fbfb-4447-b635-2a22705bc083 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:47.985198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c9ae52-fbfb-4447-b635-2a22705bc083 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:48:47.986627 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c9ae52-fbfb-4447-b635-2a22705bc083 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:48:48.005156 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 514/1014] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:48:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:48:50.652666 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156019d5-d02b-492f-8347-e5193e52a892 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:50.653514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156019d5-d02b-492f-8347-e5193e52a892 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:50.653514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156019d5-d02b-492f-8347-e5193e52a892 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:50.654598 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 501/1015] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:48:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:48:50.660021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca37548-a214-4588-bde3-1d92b5098236 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:50.660509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca37548-a214-4588-bde3-1d92b5098236 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:50.660647 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca37548-a214-4588-bde3-1d92b5098236 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:50.729941 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-cca37548-a214-4588-bde3-1d92b5098236 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:48:50.731419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-cca37548-a214-4588-bde3-1d92b5098236 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:50.740249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:50.829786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 515/1016] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:48:50 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:48:51.166990 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156019d5-d02b-492f-8347-e5193e52a892 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:48:51.169043 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-156019d5-d02b-492f-8347-e5193e52a892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:51.290626 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156019d5-d02b-492f-8347-e5193e52a892 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:48:51.376729 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf01dddf-b777-4337-813e-a5fc2d75d78c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:48:51.377408 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf01dddf-b777-4337-813e-a5fc2d75d78c service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:51.377717 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf01dddf-b777-4337-813e-a5fc2d75d78c service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:51.378020 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf01dddf-b777-4337-813e-a5fc2d75d78c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:51.379935 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-cf01dddf-b777-4337-813e-a5fc2d75d78c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:51.421408 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:51.502124 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf01dddf-b777-4337-813e-a5fc2d75d78c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:48:51.503608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf01dddf-b777-4337-813e-a5fc2d75d78c service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:48:51.523535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 502/1017] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:48:51 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:48:55.808110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5b8c30-19df-4595-b40f-8be6ca24f82e None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:55.808672 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5b8c30-19df-4595-b40f-8be6ca24f82e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:55.808672 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5b8c30-19df-4595-b40f-8be6ca24f82e None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:55.810546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 516/1018] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:48:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:48:55.816055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec924b8-11b9-48fd-82f1-85d6a928f248 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:55.816517 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec924b8-11b9-48fd-82f1-85d6a928f248 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:55.816609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec924b8-11b9-48fd-82f1-85d6a928f248 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:55.890176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-5ec924b8-11b9-48fd-82f1-85d6a928f248 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:48:55.891670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5ec924b8-11b9-48fd-82f1-85d6a928f248 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:55.992220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:56.006306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 503/1019] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:48:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:48:56.292464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5b8c30-19df-4595-b40f-8be6ca24f82e None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:48:56.295331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1d5b8c30-19df-4595-b40f-8be6ca24f82e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:56.412734 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5b8c30-19df-4595-b40f-8be6ca24f82e None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:48:56.452733 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:56.493652 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e70865-7aa3-4689-afdb-3ef0d37fc547 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:48:56.494296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e70865-7aa3-4689-afdb-3ef0d37fc547 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:56.494614 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e70865-7aa3-4689-afdb-3ef0d37fc547 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:56.494911 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e70865-7aa3-4689-afdb-3ef0d37fc547 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:56.496770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-65e70865-7aa3-4689-afdb-3ef0d37fc547 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:56.622330 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e70865-7aa3-4689-afdb-3ef0d37fc547 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:48:56.623823 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e70865-7aa3-4689-afdb-3ef0d37fc547 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:48:56.646329 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 517/1020] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:48:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:48:59.476771 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c0a759-d935-4a17-9aba-d8f7db308bed None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:59.477335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c0a759-d935-4a17-9aba-d8f7db308bed None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:59.477446 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c0a759-d935-4a17-9aba-d8f7db308bed None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:59.479123 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 504/1021] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:48:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:48:59.485915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd5937b-e000-45e6-a490-4cfda6bf45e2 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:48:59.486537 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd5937b-e000-45e6-a490-4cfda6bf45e2 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:48:59.486537 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd5937b-e000-45e6-a490-4cfda6bf45e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:48:59.561617 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-bfd5937b-e000-45e6-a490-4cfda6bf45e2 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:48:59.562842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-bfd5937b-e000-45e6-a490-4cfda6bf45e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:48:59.581473 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:48:59.666991 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 518/1022] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:48:59 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:48:59.949375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c0a759-d935-4a17-9aba-d8f7db308bed None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:48:59.951591 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-99c0a759-d935-4a17-9aba-d8f7db308bed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:00.063351 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c0a759-d935-4a17-9aba-d8f7db308bed None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:49:00.140973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283555d7-46d6-4888-9e5b-bbff2fb29a19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:49:00.141586 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283555d7-46d6-4888-9e5b-bbff2fb29a19 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:00.141864 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283555d7-46d6-4888-9e5b-bbff2fb29a19 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:00.142136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283555d7-46d6-4888-9e5b-bbff2fb29a19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:00.143756 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-283555d7-46d6-4888-9e5b-bbff2fb29a19 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:00.190079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:00.283677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283555d7-46d6-4888-9e5b-bbff2fb29a19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:49:00.285039 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283555d7-46d6-4888-9e5b-bbff2fb29a19 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:49:00.304931 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 505/1023] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:48:59 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:49:04.975070 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f73e7-d71c-4265-bb5f-4ced00b6aafd None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:04.975692 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f73e7-d71c-4265-bb5f-4ced00b6aafd None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:04.975692 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f73e7-d71c-4265-bb5f-4ced00b6aafd None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:04.977251 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 519/1024] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:49:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:04.983582 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a36598-b9fd-47a9-a6ed-966b381108b7 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:04.984159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a36598-b9fd-47a9-a6ed-966b381108b7 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:04.984459 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a36598-b9fd-47a9-a6ed-966b381108b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:05.069499 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-87a36598-b9fd-47a9-a6ed-966b381108b7 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:05.071078 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-87a36598-b9fd-47a9-a6ed-966b381108b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:05.198995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:05.202780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 506/1025] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:49:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:05.540239 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72f73e7-d71c-4265-bb5f-4ced00b6aafd None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:49:05.541812 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b72f73e7-d71c-4265-bb5f-4ced00b6aafd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:05.645388 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72f73e7-d71c-4265-bb5f-4ced00b6aafd None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:49:05.694317 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:05.723203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a52767-3237-4735-bb73-b388457002ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:49:05.723917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a52767-3237-4735-bb73-b388457002ac service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:05.724217 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a52767-3237-4735-bb73-b388457002ac service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:05.724502 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a52767-3237-4735-bb73-b388457002ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:05.726271 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-85a52767-3237-4735-bb73-b388457002ac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:05.839651 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a52767-3237-4735-bb73-b388457002ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:49:05.841170 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a52767-3237-4735-bb73-b388457002ac service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:49:05.860048 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 520/1026] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:49:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:49:10.903796 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58927c81-ba7d-4a76-a727-bbe2e816696a None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:10.904295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58927c81-ba7d-4a76-a727-bbe2e816696a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:10.904405 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58927c81-ba7d-4a76-a727-bbe2e816696a None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:10.905710 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 507/1027] 10.222.0.36 () {54 vars in 921 bytes} [Mon Oct 31 14:49:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:10.911215 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113071d9-f3f5-4109-84fe-31e3c07c5c6e None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:10.911748 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113071d9-f3f5-4109-84fe-31e3c07c5c6e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:10.912323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113071d9-f3f5-4109-84fe-31e3c07c5c6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:11.042187 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-113071d9-f3f5-4109-84fe-31e3c07c5c6e None None] MFA Rules not processed for user `70ea2aaf529b46dabb70b71f4dcc3860`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:11.044858 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-113071d9-f3f5-4109-84fe-31e3c07c5c6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:11.068951 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:11.216509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 521/1028] 10.222.0.36 () {58 vars in 1018 bytes} [Mon Oct 31 14:49:10 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:15.777478 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f8fb07-17b1-40c8-b3ea-ab159f6c3538 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:15.778625 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f8fb07-17b1-40c8-b3ea-ab159f6c3538 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:15.778625 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f8fb07-17b1-40c8-b3ea-ab159f6c3538 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:15.780121 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 508/1029] 10.222.0.36 () {54 vars in 921 bytes} [Mon Oct 31 14:49:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:15.785243 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f9c8a6-c2ba-4c0c-a733-44b3e483df9d None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:15.785722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f9c8a6-c2ba-4c0c-a733-44b3e483df9d None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:15.785808 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f9c8a6-c2ba-4c0c-a733-44b3e483df9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:15.870335 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-86f9c8a6-c2ba-4c0c-a733-44b3e483df9d None None] MFA Rules not processed for user `70ea2aaf529b46dabb70b71f4dcc3860`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:15.872283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-86f9c8a6-c2ba-4c0c-a733-44b3e483df9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:15.959101 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:16.004526 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 522/1030] 10.222.0.36 () {58 vars in 1018 bytes} [Mon Oct 31 14:49:15 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:17.532837 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428ae7f1-f2a8-43f1-8d87-01b364069cce None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:17.533337 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428ae7f1-f2a8-43f1-8d87-01b364069cce None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:17.533545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428ae7f1-f2a8-43f1-8d87-01b364069cce None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:17.534773 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 509/1031] 10.222.0.36 () {54 vars in 921 bytes} [Mon Oct 31 14:49:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:17.540186 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e95caa-bbb8-47ad-a9fb-ddababbc1a46 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:17.540589 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e95caa-bbb8-47ad-a9fb-ddababbc1a46 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:17.540822 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e95caa-bbb8-47ad-a9fb-ddababbc1a46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:17.631901 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-50e95caa-bbb8-47ad-a9fb-ddababbc1a46 None None] MFA Rules not processed for user `70ea2aaf529b46dabb70b71f4dcc3860`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:17.633127 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-50e95caa-bbb8-47ad-a9fb-ddababbc1a46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:17.743186 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 523/1032] 10.222.0.36 () {58 vars in 1018 bytes} [Mon Oct 31 14:49:17 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:21.525245 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a71a22-30ad-4d69-b173-2f12614248f9 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:21.526096 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a71a22-30ad-4d69-b173-2f12614248f9 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:21.528497 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a71a22-30ad-4d69-b173-2f12614248f9 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:21.530414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 510/1033] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:49:21 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:21.536711 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defd2f66-2815-40cb-8c5a-27b76e8dbd7c None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:21.537478 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defd2f66-2815-40cb-8c5a-27b76e8dbd7c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:21.538734 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defd2f66-2815-40cb-8c5a-27b76e8dbd7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:21.607755 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:21.643143 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-defd2f66-2815-40cb-8c5a-27b76e8dbd7c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:21.644380 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-defd2f66-2815-40cb-8c5a-27b76e8dbd7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:21.746457 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 524/1034] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:49:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:22.052390 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acba4fb3-03d9-4262-b489-126bef78decb None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:22.052572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acba4fb3-03d9-4262-b489-126bef78decb None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:22.053018 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acba4fb3-03d9-4262-b489-126bef78decb None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:22.054526 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 511/1035] 10.222.0.36 () {54 vars in 961 bytes} [Mon Oct 31 14:49:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:22.065690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f903a286-b561-465a-bb14-53de7e925b41 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:22.066775 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f903a286-b561-465a-bb14-53de7e925b41 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:22.067162 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f903a286-b561-465a-bb14-53de7e925b41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:22.154438 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-f903a286-b561-465a-bb14-53de7e925b41 None None] MFA Rules not processed for user `c8cace029a92406f850bb0a4ca07c461`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:22.155921 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-f903a286-b561-465a-bb14-53de7e925b41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:22.215617 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:22.290302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 525/1036] 10.222.0.36 () {58 vars in 1058 bytes} [Mon Oct 31 14:49:22 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:49:22.298390 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acba4fb3-03d9-4262-b489-126bef78decb None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:49:22.300134 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-acba4fb3-03d9-4262-b489-126bef78decb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:22.458117 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acba4fb3-03d9-4262-b489-126bef78decb None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:49:22.545607 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:22.590409 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb5379c-3a7b-47ff-b735-eae69b1bc7e4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:49:22.591245 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb5379c-3a7b-47ff-b735-eae69b1bc7e4 service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:22.591650 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb5379c-3a7b-47ff-b735-eae69b1bc7e4 service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:22.592018 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb5379c-3a7b-47ff-b735-eae69b1bc7e4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:22.593711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-7eb5379c-3a7b-47ff-b735-eae69b1bc7e4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:22.700605 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb5379c-3a7b-47ff-b735-eae69b1bc7e4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:49:22.702208 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb5379c-3a7b-47ff-b735-eae69b1bc7e4 service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:49:22.722197 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 512/1037] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:49:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:49:25.124937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314a51ad-4501-47fb-923f-e50514c77164 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:25.124937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314a51ad-4501-47fb-923f-e50514c77164 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:25.125753 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314a51ad-4501-47fb-923f-e50514c77164 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:25.130441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 526/1038] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:49:25 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:25.131715 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39e783-c32c-4291-b5d0-824dd8db4bf1 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:25.132174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39e783-c32c-4291-b5d0-824dd8db4bf1 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:25.132174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39e783-c32c-4291-b5d0-824dd8db4bf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:25.208624 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-8d39e783-c32c-4291-b5d0-824dd8db4bf1 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:25.209740 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-8d39e783-c32c-4291-b5d0-824dd8db4bf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:25.256513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:25.328166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 513/1039] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:49:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:49:25.617029 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314a51ad-4501-47fb-923f-e50514c77164 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:49:25.618558 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-314a51ad-4501-47fb-923f-e50514c77164 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:25.791168 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314a51ad-4501-47fb-923f-e50514c77164 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:49:25.802986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:25.925766 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9460d9c-22b8-4c6d-9505-c5f1c57b6f8e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:49:25.926374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9460d9c-22b8-4c6d-9505-c5f1c57b6f8e service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:25.926646 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9460d9c-22b8-4c6d-9505-c5f1c57b6f8e service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:25.926914 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9460d9c-22b8-4c6d-9505-c5f1c57b6f8e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:25.928576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a9460d9c-22b8-4c6d-9505-c5f1c57b6f8e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:26.024089 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9460d9c-22b8-4c6d-9505-c5f1c57b6f8e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:49:26.025616 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9460d9c-22b8-4c6d-9505-c5f1c57b6f8e service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:49:26.045312 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 527/1040] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:49:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:49:28.584125 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6540911-ab89-4c7e-8735-44ade48b9587 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:28.584725 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6540911-ab89-4c7e-8735-44ade48b9587 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:28.584725 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6540911-ab89-4c7e-8735-44ade48b9587 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:28.586087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 514/1041] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:49:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:28.593449 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c85727-b70b-42ca-aee3-2d46c5302627 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:28.593981 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c85727-b70b-42ca-aee3-2d46c5302627 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:28.593981 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c85727-b70b-42ca-aee3-2d46c5302627 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:28.610394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:28.675038 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-25c85727-b70b-42ca-aee3-2d46c5302627 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:28.676570 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-25c85727-b70b-42ca-aee3-2d46c5302627 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:28.797827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 528/1042] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:49:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:29.097980 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2327c8ea-c9fe-4366-8fa8-88896318e58c None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:29.098276 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2327c8ea-c9fe-4366-8fa8-88896318e58c None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:29.098545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2327c8ea-c9fe-4366-8fa8-88896318e58c None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:29.100214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 515/1043] 10.222.0.36 () {54 vars in 961 bytes} [Mon Oct 31 14:49:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:29.105428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef140e1c-2c51-4178-aa06-3cc2791da7fa None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:29.106349 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef140e1c-2c51-4178-aa06-3cc2791da7fa None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:29.106739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef140e1c-2c51-4178-aa06-3cc2791da7fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:29.194733 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-ef140e1c-2c51-4178-aa06-3cc2791da7fa None None] MFA Rules not processed for user `c8cace029a92406f850bb0a4ca07c461`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:29.196347 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ef140e1c-2c51-4178-aa06-3cc2791da7fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:29.235057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:29.325599 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 529/1044] 10.222.0.36 () {58 vars in 1058 bytes} [Mon Oct 31 14:49:29 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:29.334111 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2327c8ea-c9fe-4366-8fa8-88896318e58c None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:49:29.335797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2327c8ea-c9fe-4366-8fa8-88896318e58c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:29.494668 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2327c8ea-c9fe-4366-8fa8-88896318e58c None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:49:29.516701 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:29.623462 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62db3eef-e356-496c-b9b1-62f14d5b6b4b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:49:29.624189 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62db3eef-e356-496c-b9b1-62f14d5b6b4b service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:29.624526 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62db3eef-e356-496c-b9b1-62f14d5b6b4b service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:29.624896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62db3eef-e356-496c-b9b1-62f14d5b6b4b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:29.627199 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-62db3eef-e356-496c-b9b1-62f14d5b6b4b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:29.723641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62db3eef-e356-496c-b9b1-62f14d5b6b4b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:49:29.725413 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62db3eef-e356-496c-b9b1-62f14d5b6b4b service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:49:29.745217 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 516/1045] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:49:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:49:29.894603 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5817067d-eeb7-4b87-af87-60124aa763aa None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:29.895159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5817067d-eeb7-4b87-af87-60124aa763aa None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:29.895159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5817067d-eeb7-4b87-af87-60124aa763aa None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:29.896655 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 530/1046] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:49:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:29.902349 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968801c0-3acb-4ef1-b3aa-514ed73ec19c None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:29.902666 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968801c0-3acb-4ef1-b3aa-514ed73ec19c None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:29.903488 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968801c0-3acb-4ef1-b3aa-514ed73ec19c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:29.942740 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:29.986763 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-968801c0-3acb-4ef1-b3aa-514ed73ec19c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:29.988166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-968801c0-3acb-4ef1-b3aa-514ed73ec19c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:30.102678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 517/1047] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:49:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:30.436397 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790df32d-4290-4bd7-b4dc-673b12683c24 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:30.436397 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790df32d-4290-4bd7-b4dc-673b12683c24 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:30.436397 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790df32d-4290-4bd7-b4dc-673b12683c24 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:30.437980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 531/1048] 10.222.0.36 () {54 vars in 963 bytes} [Mon Oct 31 14:49:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:30.442476 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0b5694-798d-475a-abe8-8786cc51eae8 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:30.442476 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0b5694-798d-475a-abe8-8786cc51eae8 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:30.442476 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0b5694-798d-475a-abe8-8786cc51eae8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:30.524504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-6f0b5694-798d-475a-abe8-8786cc51eae8 None None] MFA Rules not processed for user `993d553a46044c18a4eec6e408c0969a`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:30.525834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6f0b5694-798d-475a-abe8-8786cc51eae8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:30.579781 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:30.612372 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 518/1049] 10.222.0.36 () {58 vars in 1060 bytes} [Mon Oct 31 14:49:30 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:30.619909 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790df32d-4290-4bd7-b4dc-673b12683c24 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:49:30.622044 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-790df32d-4290-4bd7-b4dc-673b12683c24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:30.740588 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790df32d-4290-4bd7-b4dc-673b12683c24 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:49:30.783740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:30.820669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dab49cc-592e-470e-b7e7-a2f96b676265 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:49:30.821420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dab49cc-592e-470e-b7e7-a2f96b676265 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:30.821822 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dab49cc-592e-470e-b7e7-a2f96b676265 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:30.822192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dab49cc-592e-470e-b7e7-a2f96b676265 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:30.823980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1dab49cc-592e-470e-b7e7-a2f96b676265 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:30.967121 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dab49cc-592e-470e-b7e7-a2f96b676265 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:49:30.968899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dab49cc-592e-470e-b7e7-a2f96b676265 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:49:30.990112 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 532/1050] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:49:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:49:32.016883 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028ba5e3-d4f5-454d-bc79-ee31be772c13 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:32.017425 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028ba5e3-d4f5-454d-bc79-ee31be772c13 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:32.017425 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028ba5e3-d4f5-454d-bc79-ee31be772c13 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:32.018783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 519/1051] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:49:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:32.024555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cac8300-c440-4f1a-9558-3a0eb20e933e None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:32.025078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cac8300-c440-4f1a-9558-3a0eb20e933e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:32.025177 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cac8300-c440-4f1a-9558-3a0eb20e933e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:32.099993 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-5cac8300-c440-4f1a-9558-3a0eb20e933e None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:32.101224 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5cac8300-c440-4f1a-9558-3a0eb20e933e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:32.122009 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:32.197525 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 533/1052] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:49:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:32.490682 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028ba5e3-d4f5-454d-bc79-ee31be772c13 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:49:32.492322 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-028ba5e3-d4f5-454d-bc79-ee31be772c13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:32.661038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028ba5e3-d4f5-454d-bc79-ee31be772c13 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:49:32.716532 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:32.784541 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58573090-4187-4954-b1a4-af60e1e38871 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:49:32.785151 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58573090-4187-4954-b1a4-af60e1e38871 service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:32.785422 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58573090-4187-4954-b1a4-af60e1e38871 service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:32.785687 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58573090-4187-4954-b1a4-af60e1e38871 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:32.787280 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-58573090-4187-4954-b1a4-af60e1e38871 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:32.883680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58573090-4187-4954-b1a4-af60e1e38871 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:49:32.885767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58573090-4187-4954-b1a4-af60e1e38871 service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:49:32.905825 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 520/1053] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:49:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:49:35.148481 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a30880-09ef-4f68-bc37-6839494437ed None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:35.149017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a30880-09ef-4f68-bc37-6839494437ed None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:35.149017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a30880-09ef-4f68-bc37-6839494437ed None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:35.150338 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 534/1054] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:49:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:35.156384 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5284701e-62dd-4fc9-b7a2-4ed252df79d5 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:35.156805 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5284701e-62dd-4fc9-b7a2-4ed252df79d5 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:35.156984 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5284701e-62dd-4fc9-b7a2-4ed252df79d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:35.229151 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-5284701e-62dd-4fc9-b7a2-4ed252df79d5 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:35.230437 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5284701e-62dd-4fc9-b7a2-4ed252df79d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:35.238098 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:35.344649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 521/1055] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:49:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:35.645126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a30880-09ef-4f68-bc37-6839494437ed None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:49:35.647179 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-86a30880-09ef-4f68-bc37-6839494437ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:35.800938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a30880-09ef-4f68-bc37-6839494437ed None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:49:35.804842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:35.919933 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec0873f-24be-43c9-a1ae-2baf378f33bd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:49:35.920546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec0873f-24be-43c9-a1ae-2baf378f33bd service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:35.920824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec0873f-24be-43c9-a1ae-2baf378f33bd service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:35.921096 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec0873f-24be-43c9-a1ae-2baf378f33bd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:35.922680 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-6ec0873f-24be-43c9-a1ae-2baf378f33bd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:36.021441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec0873f-24be-43c9-a1ae-2baf378f33bd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:49:36.023150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec0873f-24be-43c9-a1ae-2baf378f33bd service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:49:36.042189 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 535/1056] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:49:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:49:38.274039 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769861b9-1d94-4078-8f43-00d6a2625ab4 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:38.274602 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769861b9-1d94-4078-8f43-00d6a2625ab4 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:38.274882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769861b9-1d94-4078-8f43-00d6a2625ab4 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:38.276531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 522/1057] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:49:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:38.282140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8961922-587a-42fa-b193-e70eaadd8248 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:38.282546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8961922-587a-42fa-b193-e70eaadd8248 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:38.282768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8961922-587a-42fa-b193-e70eaadd8248 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:38.294629 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:38.357427 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-c8961922-587a-42fa-b193-e70eaadd8248 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:38.358847 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-c8961922-587a-42fa-b193-e70eaadd8248 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:38.503725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 536/1058] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:49:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:38.799356 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769861b9-1d94-4078-8f43-00d6a2625ab4 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:49:38.801012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-769861b9-1d94-4078-8f43-00d6a2625ab4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:38.960025 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:38.977206 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769861b9-1d94-4078-8f43-00d6a2625ab4 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:49:39.117798 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bc331c-d50a-40b2-9143-1d2a02559ed7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:49:39.118613 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bc331c-d50a-40b2-9143-1d2a02559ed7 service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:39.119039 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bc331c-d50a-40b2-9143-1d2a02559ed7 service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:39.119424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bc331c-d50a-40b2-9143-1d2a02559ed7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:39.122636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-07bc331c-d50a-40b2-9143-1d2a02559ed7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:39.227511 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bc331c-d50a-40b2-9143-1d2a02559ed7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:49:39.229213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bc331c-d50a-40b2-9143-1d2a02559ed7 service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:49:39.240256 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:39.250707 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 523/1059] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:49:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:49:41.853442 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eb9863-7ac6-455c-b7a0-a326e67d6cd1 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:41.853955 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eb9863-7ac6-455c-b7a0-a326e67d6cd1 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:41.853955 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eb9863-7ac6-455c-b7a0-a326e67d6cd1 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:41.855260 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 537/1060] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:49:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:41.861434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6091b984-01cb-4038-be31-34bf64894735 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:41.861883 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6091b984-01cb-4038-be31-34bf64894735 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:41.861992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6091b984-01cb-4038-be31-34bf64894735 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:41.931338 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-6091b984-01cb-4038-be31-34bf64894735 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:41.932678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6091b984-01cb-4038-be31-34bf64894735 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:42.036296 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 524/1061] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:49:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:42.522984 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73eb9863-7ac6-455c-b7a0-a326e67d6cd1 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:49:42.524583 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-73eb9863-7ac6-455c-b7a0-a326e67d6cd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:42.672844 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:42.740688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73eb9863-7ac6-455c-b7a0-a326e67d6cd1 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:49:42.868436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fead74c1-bf4a-43a1-89eb-7b6e538de7c7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:49:42.869215 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fead74c1-bf4a-43a1-89eb-7b6e538de7c7 service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:42.869402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fead74c1-bf4a-43a1-89eb-7b6e538de7c7 service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:42.869656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fead74c1-bf4a-43a1-89eb-7b6e538de7c7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:42.871274 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-fead74c1-bf4a-43a1-89eb-7b6e538de7c7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:42.929984 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:42.975013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fead74c1-bf4a-43a1-89eb-7b6e538de7c7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:49:42.976882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fead74c1-bf4a-43a1-89eb-7b6e538de7c7 service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:49:42.998414 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 538/1062] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:49:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 476 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:49:45.575576 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3749ce74-2826-4a09-8c3b-d758af141feb None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:45.576139 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3749ce74-2826-4a09-8c3b-d758af141feb None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:45.576139 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3749ce74-2826-4a09-8c3b-d758af141feb None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:45.577472 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 525/1063] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:49:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:45.582653 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810071fc-87ad-4d05-9799-21126ef8cd3d None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:45.583066 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810071fc-87ad-4d05-9799-21126ef8cd3d None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:45.583195 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810071fc-87ad-4d05-9799-21126ef8cd3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:45.649961 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-810071fc-87ad-4d05-9799-21126ef8cd3d None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:45.651166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-810071fc-87ad-4d05-9799-21126ef8cd3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:45.749647 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 539/1064] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:49:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:46.044939 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3749ce74-2826-4a09-8c3b-d758af141feb None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:49:46.046447 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-3749ce74-2826-4a09-8c3b-d758af141feb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:46.127272 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:46.200904 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3749ce74-2826-4a09-8c3b-d758af141feb None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:49:46.316135 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc9502d-a8fe-4824-acb6-1968a6bd9d1b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:49:46.316660 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc9502d-a8fe-4824-acb6-1968a6bd9d1b service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:46.316940 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc9502d-a8fe-4824-acb6-1968a6bd9d1b service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:46.317215 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc9502d-a8fe-4824-acb6-1968a6bd9d1b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:46.318815 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-9cc9502d-a8fe-4824-acb6-1968a6bd9d1b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:46.363044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:46.416143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc9502d-a8fe-4824-acb6-1968a6bd9d1b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:49:46.417731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc9502d-a8fe-4824-acb6-1968a6bd9d1b service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:49:46.437418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 526/1065] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:49:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:49:46.459309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed0f6db-7dda-47b9-8d27-a4e7969b3e07 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:46.459707 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed0f6db-7dda-47b9-8d27-a4e7969b3e07 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:46.460081 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed0f6db-7dda-47b9-8d27-a4e7969b3e07 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:46.461481 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 540/1066] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:49:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:46.467656 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1722db-6f96-46d3-b6ab-07fbc4c57de3 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:46.467941 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1722db-6f96-46d3-b6ab-07fbc4c57de3 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:46.468278 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1722db-6f96-46d3-b6ab-07fbc4c57de3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:46.552338 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-2e1722db-6f96-46d3-b6ab-07fbc4c57de3 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:46.553621 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2e1722db-6f96-46d3-b6ab-07fbc4c57de3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:46.659597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 527/1067] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:49:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:46.963063 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3725e090-ed8c-4177-ade1-572abc3c0075 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:46.963883 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3725e090-ed8c-4177-ade1-572abc3c0075 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:46.963998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3725e090-ed8c-4177-ade1-572abc3c0075 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:46.965836 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 541/1068] 10.222.0.36 () {54 vars in 962 bytes} [Mon Oct 31 14:49:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:46.971698 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f395c6a2-1ba8-4dc3-abe1-5644fb7d4880 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:46.972009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f395c6a2-1ba8-4dc3-abe1-5644fb7d4880 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:46.972286 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f395c6a2-1ba8-4dc3-abe1-5644fb7d4880 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:46.989514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:47.052845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-f395c6a2-1ba8-4dc3-abe1-5644fb7d4880 None None] MFA Rules not processed for user `a9808dcba0b64027bb70e31d1569957d`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:47.054044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-f395c6a2-1ba8-4dc3-abe1-5644fb7d4880 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:47.132713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 528/1069] 10.222.0.36 () {58 vars in 1059 bytes} [Mon Oct 31 14:49:46 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:47.141173 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3725e090-ed8c-4177-ade1-572abc3c0075 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:49:47.142666 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-3725e090-ed8c-4177-ade1-572abc3c0075 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:47.174679 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:47.250079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3725e090-ed8c-4177-ade1-572abc3c0075 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:49:47.322937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b5737c-f169-43f0-9de8-d6f8495a2643 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9808dcba0b64027bb70e31d1569957d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:49:47.323564 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b5737c-f169-43f0-9de8-d6f8495a2643 service glance] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:47.323849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b5737c-f169-43f0-9de8-d6f8495a2643 service glance] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:47.324119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b5737c-f169-43f0-9de8-d6f8495a2643 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:47.326264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-17b5737c-f169-43f0-9de8-d6f8495a2643 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:47.407832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:47.440727 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b5737c-f169-43f0-9de8-d6f8495a2643 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:49:47.442241 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b5737c-f169-43f0-9de8-d6f8495a2643 service glance] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:49:47.462394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 542/1070] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:49:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:49:48.216856 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da20361e-1854-414d-a0c5-816bfcc3346e None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:48.217369 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da20361e-1854-414d-a0c5-816bfcc3346e None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:48.217472 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da20361e-1854-414d-a0c5-816bfcc3346e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:48.287461 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-da20361e-1854-414d-a0c5-816bfcc3346e None None] MFA Rules not processed for user `e0d0f8598fac40449bcb2a55ff8df8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:48.288701 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-da20361e-1854-414d-a0c5-816bfcc3346e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:48.346936 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:48.379148 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 529/1071] 10.222.0.36 () {58 vars in 1009 bytes} [Mon Oct 31 14:49:48 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:48.426348 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f65ade-c834-4c58-9665-e3b00eb07cf1 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:48.426835 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f65ade-c834-4c58-9665-e3b00eb07cf1 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:48.426962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f65ade-c834-4c58-9665-e3b00eb07cf1 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:48.428287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 543/1072] 10.222.0.36 () {54 vars in 943 bytes} [Mon Oct 31 14:49:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:48.435164 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5059fd26-6c77-43de-8aa8-874ec5746d7c None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:48.435454 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5059fd26-6c77-43de-8aa8-874ec5746d7c None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:48.435772 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5059fd26-6c77-43de-8aa8-874ec5746d7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:48.513910 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-5059fd26-6c77-43de-8aa8-874ec5746d7c None None] MFA Rules not processed for user `f4ae5e20cc5b40ccbdb951f908f5080d`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:48.515155 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5059fd26-6c77-43de-8aa8-874ec5746d7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:48.630550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 530/1073] 10.222.0.36 () {58 vars in 1040 bytes} [Mon Oct 31 14:49:48 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:49:48.639780 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f65ade-c834-4c58-9665-e3b00eb07cf1 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:49:48.641242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a5f65ade-c834-4c58-9665-e3b00eb07cf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:48.682638 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de054806-2d64-4796-9628-e438c7de2634 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:48.682922 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de054806-2d64-4796-9628-e438c7de2634 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:48.683225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de054806-2d64-4796-9628-e438c7de2634 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:48.685424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 531/1074] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:49:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:49:48.690719 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8defa6-c550-401d-a14b-3d4c9d36a907 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:48.691002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8defa6-c550-401d-a14b-3d4c9d36a907 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:48.691278 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8defa6-c550-401d-a14b-3d4c9d36a907 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:48.732057 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:48.762954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-fc8defa6-c550-401d-a14b-3d4c9d36a907 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:48.764202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-fc8defa6-c550-401d-a14b-3d4c9d36a907 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:48.788245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f65ade-c834-4c58-9665-e3b00eb07cf1 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:49:48.838886 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:48.866381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 532/1075] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:49:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:49:48.899978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1224d8c-f6ab-4b86-a55a-18174be9074f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4ae5e20cc5b40ccbdb951f908f5080d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:49:48.900499 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1224d8c-f6ab-4b86-a55a-18174be9074f service swift] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:48.900772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1224d8c-f6ab-4b86-a55a-18174be9074f service swift] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:48.901041 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1224d8c-f6ab-4b86-a55a-18174be9074f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:48.903150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e1224d8c-f6ab-4b86-a55a-18174be9074f service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:48.985803 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1224d8c-f6ab-4b86-a55a-18174be9074f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:49:48.987376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1224d8c-f6ab-4b86-a55a-18174be9074f service swift] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:49:48.995419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 544/1076] 10.222.0.36 () {60 vars in 1384 bytes} [Mon Oct 31 14:49:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 31 14:49:49.161167 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8defa6-c550-401d-a14b-3d4c9d36a907 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:49:49.162403 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-fc8defa6-c550-401d-a14b-3d4c9d36a907 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:49.307233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:49.311724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8defa6-c550-401d-a14b-3d4c9d36a907 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:49:49.440353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7320108e-877c-474b-9185-f58f95a014fe service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:49:49.440881 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7320108e-877c-474b-9185-f58f95a014fe service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:49.441159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7320108e-877c-474b-9185-f58f95a014fe service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:49.441431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7320108e-877c-474b-9185-f58f95a014fe service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:49.443110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-7320108e-877c-474b-9185-f58f95a014fe service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:49.544142 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7320108e-877c-474b-9185-f58f95a014fe service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:49:49.545822 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7320108e-877c-474b-9185-f58f95a014fe service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:49:49.565762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 533/1077] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:49:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:49:51.783184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a5aa97-6018-4e43-aaa0-1f66c4e58327 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:51.783830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a5aa97-6018-4e43-aaa0-1f66c4e58327 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:51.783830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a5aa97-6018-4e43-aaa0-1f66c4e58327 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:51.789199 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 545/1078] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:49:51 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:51.790857 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d21c10-0989-4d33-9365-6858d449c2b8 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:51.791309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d21c10-0989-4d33-9365-6858d449c2b8 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:51.791399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d21c10-0989-4d33-9365-6858d449c2b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:51.797591 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:51.878205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-a9d21c10-0989-4d33-9365-6858d449c2b8 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:51.879719 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a9d21c10-0989-4d33-9365-6858d449c2b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:51.998479 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 534/1079] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:49:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:52.289969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a5aa97-6018-4e43-aaa0-1f66c4e58327 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:49:52.291766 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-54a5aa97-6018-4e43-aaa0-1f66c4e58327 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:52.411531 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:52.484760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a5aa97-6018-4e43-aaa0-1f66c4e58327 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:49:52.619062 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254c26a7-4697-46a1-894a-81e0ede03d87 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:49:52.619814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254c26a7-4697-46a1-894a-81e0ede03d87 service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:52.620160 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254c26a7-4697-46a1-894a-81e0ede03d87 service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:52.620499 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254c26a7-4697-46a1-894a-81e0ede03d87 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:52.622708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-254c26a7-4697-46a1-894a-81e0ede03d87 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:52.668776 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:52.723808 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254c26a7-4697-46a1-894a-81e0ede03d87 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:49:52.725678 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254c26a7-4697-46a1-894a-81e0ede03d87 service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:49:52.744712 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 546/1080] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:49:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 455 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:49:54.978882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24521124-e29b-418e-83bf-192a3f05d72b None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:54.979414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24521124-e29b-418e-83bf-192a3f05d72b None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:54.979414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24521124-e29b-418e-83bf-192a3f05d72b None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:54.981356 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 535/1081] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:49:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:54.986647 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e50c12-e159-4bbc-971f-6cea1f006a0c None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:54.987071 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e50c12-e159-4bbc-971f-6cea1f006a0c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:54.987196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e50c12-e159-4bbc-971f-6cea1f006a0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:55.058569 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-67e50c12-e159-4bbc-971f-6cea1f006a0c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:55.059855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-67e50c12-e159-4bbc-971f-6cea1f006a0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:55.166155 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 547/1082] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:49:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:49:55.461888 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24521124-e29b-418e-83bf-192a3f05d72b None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:49:55.463402 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-24521124-e29b-418e-83bf-192a3f05d72b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:55.556888 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:55.627903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24521124-e29b-418e-83bf-192a3f05d72b None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:49:55.757552 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e57e7a-35be-446a-ac8f-bdcbc17fa53f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:49:55.758070 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e57e7a-35be-446a-ac8f-bdcbc17fa53f service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:55.758343 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e57e7a-35be-446a-ac8f-bdcbc17fa53f service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:55.758607 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e57e7a-35be-446a-ac8f-bdcbc17fa53f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:55.760229 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-04e57e7a-35be-446a-ac8f-bdcbc17fa53f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:55.809919 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:55.860414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e57e7a-35be-446a-ac8f-bdcbc17fa53f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:49:55.862125 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e57e7a-35be-446a-ac8f-bdcbc17fa53f service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:49:55.881049 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 536/1083] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:49:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:49:58.098461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0818e966-1e2c-4e9f-9897-082a0ab30c11 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:58.098984 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0818e966-1e2c-4e9f-9897-082a0ab30c11 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:58.098984 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0818e966-1e2c-4e9f-9897-082a0ab30c11 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:58.102295 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 548/1084] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:49:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:49:58.107848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0874a9-f1c7-4648-97ac-aa7e0d772b73 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:58.107848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0874a9-f1c7-4648-97ac-aa7e0d772b73 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:58.107848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0874a9-f1c7-4648-97ac-aa7e0d772b73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:58.186559 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-4f0874a9-f1c7-4648-97ac-aa7e0d772b73 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:49:58.187825 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-4f0874a9-f1c7-4648-97ac-aa7e0d772b73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:58.301769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 537/1085] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:49:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:49:58.592767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0818e966-1e2c-4e9f-9897-082a0ab30c11 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:49:58.594294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0818e966-1e2c-4e9f-9897-082a0ab30c11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:58.617775 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:58.755828 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0818e966-1e2c-4e9f-9897-082a0ab30c11 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:49:58.853841 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:49:58.875333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab59de7-239b-4502-b22b-5bb3c9aae7ec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:49:58.875971 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab59de7-239b-4502-b22b-5bb3c9aae7ec service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:49:58.876245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab59de7-239b-4502-b22b-5bb3c9aae7ec service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:49:58.876514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab59de7-239b-4502-b22b-5bb3c9aae7ec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:49:58.878094 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-aab59de7-239b-4502-b22b-5bb3c9aae7ec service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:49:58.977152 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab59de7-239b-4502-b22b-5bb3c9aae7ec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:49:58.978734 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab59de7-239b-4502-b22b-5bb3c9aae7ec service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:49:58.999758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 549/1086] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:49:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:50:08.300274 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=91 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:08.997156 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=53 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:12.307395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db22616a-6357-453e-9e5f-d348eabd0329 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:12.307999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db22616a-6357-453e-9e5f-d348eabd0329 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:12.307999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db22616a-6357-453e-9e5f-d348eabd0329 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:12.309410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 538/1087] 10.222.0.36 () {54 vars in 918 bytes} [Mon Oct 31 14:50:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:12.314623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acfe069-8ef7-4702-ab39-3719347c2469 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:12.315065 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acfe069-8ef7-4702-ab39-3719347c2469 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:12.315176 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acfe069-8ef7-4702-ab39-3719347c2469 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:12.390772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-3acfe069-8ef7-4702-ab39-3719347c2469 None None] MFA Rules not processed for user `70ea2aaf529b46dabb70b71f4dcc3860`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:12.392010 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-3acfe069-8ef7-4702-ab39-3719347c2469 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:12.496147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 550/1088] 10.222.0.36 () {58 vars in 1015 bytes} [Mon Oct 31 14:50:12 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:12.510682 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5b9c1d-2b8f-4358-9392-972f9cb27908 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:12.510969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5b9c1d-2b8f-4358-9392-972f9cb27908 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:12.511244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5b9c1d-2b8f-4358-9392-972f9cb27908 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:12.513617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 539/1089] 10.222.0.36 () {54 vars in 918 bytes} [Mon Oct 31 14:50:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:12.518094 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba28d29d-0a66-4ea8-a1bf-561cab089943 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:12.518389 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba28d29d-0a66-4ea8-a1bf-561cab089943 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:12.518659 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba28d29d-0a66-4ea8-a1bf-561cab089943 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:12.591920 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:12.594866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-ba28d29d-0a66-4ea8-a1bf-561cab089943 None None] MFA Rules not processed for user `c8cace029a92406f850bb0a4ca07c461`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:12.596589 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ba28d29d-0a66-4ea8-a1bf-561cab089943 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:12.710862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 551/1090] 10.222.0.36 () {58 vars in 1015 bytes} [Mon Oct 31 14:50:12 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:15.355925 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a215fe5a-a561-40ca-a8e5-7a58a45624ad None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:15.356459 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a215fe5a-a561-40ca-a8e5-7a58a45624ad None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:15.356587 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a215fe5a-a561-40ca-a8e5-7a58a45624ad None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:15.357996 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 540/1091] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 31 14:50:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:15.364126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978df696-a060-4a07-ada8-b201f5d82d34 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:15.364633 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978df696-a060-4a07-ada8-b201f5d82d34 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:15.365110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978df696-a060-4a07-ada8-b201f5d82d34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:15.441204 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-978df696-a060-4a07-ada8-b201f5d82d34 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:15.442451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-978df696-a060-4a07-ada8-b201f5d82d34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:15.507943 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:15.560858 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 552/1092] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 31 14:50:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:15.849631 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a215fe5a-a561-40ca-a8e5-7a58a45624ad None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:15.851217 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a215fe5a-a561-40ca-a8e5-7a58a45624ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:15.999440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a215fe5a-a561-40ca-a8e5-7a58a45624ad None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:16.072110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:16.114225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cf33a7-346c-48ee-8b90-918a377dd17d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:16.114751 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cf33a7-346c-48ee-8b90-918a377dd17d service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:16.115038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cf33a7-346c-48ee-8b90-918a377dd17d service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:16.115308 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cf33a7-346c-48ee-8b90-918a377dd17d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:16.116952 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-33cf33a7-346c-48ee-8b90-918a377dd17d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:16.225807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cf33a7-346c-48ee-8b90-918a377dd17d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:16.227395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cf33a7-346c-48ee-8b90-918a377dd17d service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:16.246833 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 541/1093] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:50:17.388530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fd6012-b9a7-419c-82e5-65fda56c5396 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:17.389091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fd6012-b9a7-419c-82e5-65fda56c5396 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:17.389091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fd6012-b9a7-419c-82e5-65fda56c5396 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:17.390457 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 553/1094] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 31 14:50:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:17.395465 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5799630-6031-4ec9-be82-c1d76de30b0c None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:17.395918 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5799630-6031-4ec9-be82-c1d76de30b0c None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:17.396064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5799630-6031-4ec9-be82-c1d76de30b0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:17.475796 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-e5799630-6031-4ec9-be82-c1d76de30b0c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:17.477034 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e5799630-6031-4ec9-be82-c1d76de30b0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:17.503978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:17.577698 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 542/1095] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 31 14:50:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:17.737337 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fd6012-b9a7-419c-82e5-65fda56c5396 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:17.738805 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-37fd6012-b9a7-419c-82e5-65fda56c5396 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:17.880378 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fd6012-b9a7-419c-82e5-65fda56c5396 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:17.922791 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:17.996912 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbeaba29-ebc3-4747-9f28-281e95adc6a1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:17.997433 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbeaba29-ebc3-4747-9f28-281e95adc6a1 service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:17.997713 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbeaba29-ebc3-4747-9f28-281e95adc6a1 service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:17.997983 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbeaba29-ebc3-4747-9f28-281e95adc6a1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:17.999617 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-fbeaba29-ebc3-4747-9f28-281e95adc6a1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:18.088773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbeaba29-ebc3-4747-9f28-281e95adc6a1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:18.090758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbeaba29-ebc3-4747-9f28-281e95adc6a1 service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:18.109048 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 554/1096] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:50:19.227250 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15620e8-3355-4f31-9c0b-bcbffb01c5aa None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:19.227846 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15620e8-3355-4f31-9c0b-bcbffb01c5aa None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:19.227846 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15620e8-3355-4f31-9c0b-bcbffb01c5aa None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:19.229504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 543/1097] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 31 14:50:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:19.234645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b24fe4-5762-48dd-a657-6eab72faccbc None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:19.235072 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b24fe4-5762-48dd-a657-6eab72faccbc None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:19.235191 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b24fe4-5762-48dd-a657-6eab72faccbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:19.288296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:19.310755 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-82b24fe4-5762-48dd-a657-6eab72faccbc None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:19.311974 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-82b24fe4-5762-48dd-a657-6eab72faccbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:19.407763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 555/1098] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 31 14:50:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:19.560026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15620e8-3355-4f31-9c0b-bcbffb01c5aa None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:19.561472 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d15620e8-3355-4f31-9c0b-bcbffb01c5aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:19.705833 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15620e8-3355-4f31-9c0b-bcbffb01c5aa None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:19.718907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:19.817205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b89b37d-6ac2-4304-9041-6149c19d413d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:19.817716 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b89b37d-6ac2-4304-9041-6149c19d413d service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:19.818270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b89b37d-6ac2-4304-9041-6149c19d413d service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:19.818544 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b89b37d-6ac2-4304-9041-6149c19d413d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:19.820157 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6b89b37d-6ac2-4304-9041-6149c19d413d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:19.913328 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b89b37d-6ac2-4304-9041-6149c19d413d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:19.914907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b89b37d-6ac2-4304-9041-6149c19d413d service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:19.934364 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 544/1099] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:50:21.041148 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5928aa6b-f49b-466e-bf0d-33130e4d0873 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:21.041660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5928aa6b-f49b-466e-bf0d-33130e4d0873 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:21.041660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5928aa6b-f49b-466e-bf0d-33130e4d0873 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:21.043725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 556/1100] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 31 14:50:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:21.048635 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8ac0d6-4220-4fcb-9726-f9d6dcc50d7e None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:21.049067 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8ac0d6-4220-4fcb-9726-f9d6dcc50d7e None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:21.049189 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8ac0d6-4220-4fcb-9726-f9d6dcc50d7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:21.070388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:21.128537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-5b8ac0d6-4220-4fcb-9726-f9d6dcc50d7e None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:21.129764 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5b8ac0d6-4220-4fcb-9726-f9d6dcc50d7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:21.225808 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 545/1101] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 31 14:50:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:21.376999 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5928aa6b-f49b-466e-bf0d-33130e4d0873 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:21.378438 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5928aa6b-f49b-466e-bf0d-33130e4d0873 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:21.489064 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:21.524963 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5928aa6b-f49b-466e-bf0d-33130e4d0873 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:21.637596 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4744e84-233f-4067-8015-a40e36ac1993 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:21.638339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4744e84-233f-4067-8015-a40e36ac1993 service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:21.638704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4744e84-233f-4067-8015-a40e36ac1993 service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:21.639066 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4744e84-233f-4067-8015-a40e36ac1993 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:21.640961 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d4744e84-233f-4067-8015-a40e36ac1993 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:21.718525 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:21.732407 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4744e84-233f-4067-8015-a40e36ac1993 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:21.734022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4744e84-233f-4067-8015-a40e36ac1993 service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:21.751879 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 557/1102] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:50:22.849564 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ca9f17-00b6-462d-8019-6ceb9f0eca38 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:22.850125 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ca9f17-00b6-462d-8019-6ceb9f0eca38 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:22.850125 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ca9f17-00b6-462d-8019-6ceb9f0eca38 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:22.851418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 546/1103] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 31 14:50:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:22.856648 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccbf2f0-0ea0-475d-897d-b007e059dcc9 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:22.857079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccbf2f0-0ea0-475d-897d-b007e059dcc9 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:22.857227 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccbf2f0-0ea0-475d-897d-b007e059dcc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:22.933326 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-6ccbf2f0-0ea0-475d-897d-b007e059dcc9 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:22.935025 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-6ccbf2f0-0ea0-475d-897d-b007e059dcc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:23.030892 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 558/1104] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 31 14:50:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:23.181842 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ca9f17-00b6-462d-8019-6ceb9f0eca38 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:23.183361 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-19ca9f17-00b6-462d-8019-6ceb9f0eca38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:23.268734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:23.330245 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ca9f17-00b6-462d-8019-6ceb9f0eca38 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:23.441594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a47ac0-e6b1-44dd-a352-dfbb3141a495 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:23.442217 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a47ac0-e6b1-44dd-a352-dfbb3141a495 service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:23.442493 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a47ac0-e6b1-44dd-a352-dfbb3141a495 service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:23.442764 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a47ac0-e6b1-44dd-a352-dfbb3141a495 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:23.444388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-21a47ac0-e6b1-44dd-a352-dfbb3141a495 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:23.492677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:23.535744 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a47ac0-e6b1-44dd-a352-dfbb3141a495 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:23.537486 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a47ac0-e6b1-44dd-a352-dfbb3141a495 service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:23.557069 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 547/1105] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:50:24.602715 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14db3532-0065-4c6b-839d-12fc38c9809b None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:24.603272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14db3532-0065-4c6b-839d-12fc38c9809b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:24.603272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14db3532-0065-4c6b-839d-12fc38c9809b None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:24.604636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 559/1106] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 31 14:50:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:24.610398 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc19439-ac68-4b68-8af1-522048973fa1 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:24.610852 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc19439-ac68-4b68-8af1-522048973fa1 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:24.610947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc19439-ac68-4b68-8af1-522048973fa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:24.687181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-ecc19439-ac68-4b68-8af1-522048973fa1 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:24.688454 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ecc19439-ac68-4b68-8af1-522048973fa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:24.783536 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 548/1107] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 31 14:50:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:50:24.935400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14db3532-0065-4c6b-839d-12fc38c9809b None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:24.936949 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-14db3532-0065-4c6b-839d-12fc38c9809b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:24.986276 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:25.080673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14db3532-0065-4c6b-839d-12fc38c9809b None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:25.194331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20802c1c-a59a-4068-aa4f-6d9cb4cd064f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:25.194932 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20802c1c-a59a-4068-aa4f-6d9cb4cd064f service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:25.195204 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20802c1c-a59a-4068-aa4f-6d9cb4cd064f service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:25.195468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20802c1c-a59a-4068-aa4f-6d9cb4cd064f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:25.197063 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-20802c1c-a59a-4068-aa4f-6d9cb4cd064f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:25.204306 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:25.288630 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20802c1c-a59a-4068-aa4f-6d9cb4cd064f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:25.290219 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20802c1c-a59a-4068-aa4f-6d9cb4cd064f service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:25.308050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 560/1108] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:50:26.348185 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f48a60-ef15-4b4a-bcc3-5d9643eb7937 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:26.348727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f48a60-ef15-4b4a-bcc3-5d9643eb7937 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:26.348826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f48a60-ef15-4b4a-bcc3-5d9643eb7937 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:26.350138 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 549/1109] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 31 14:50:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:26.355049 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f98eab0-e888-4f7c-8776-c307b9095a13 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:26.356028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f98eab0-e888-4f7c-8776-c307b9095a13 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:26.356164 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f98eab0-e888-4f7c-8776-c307b9095a13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:26.431699 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-8f98eab0-e888-4f7c-8776-c307b9095a13 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:26.432909 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8f98eab0-e888-4f7c-8776-c307b9095a13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:26.501403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:26.531509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 561/1110] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 31 14:50:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:26.679892 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f48a60-ef15-4b4a-bcc3-5d9643eb7937 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:26.681418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-43f48a60-ef15-4b4a-bcc3-5d9643eb7937 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:26.699910 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:26.832652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f48a60-ef15-4b4a-bcc3-5d9643eb7937 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:26.918932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:26.945763 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234a44e7-c0e9-4f13-b2e4-8713fdeb8ac1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:26.946281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234a44e7-c0e9-4f13-b2e4-8713fdeb8ac1 service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:26.946555 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234a44e7-c0e9-4f13-b2e4-8713fdeb8ac1 service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:26.946823 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234a44e7-c0e9-4f13-b2e4-8713fdeb8ac1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:26.948455 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-234a44e7-c0e9-4f13-b2e4-8713fdeb8ac1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:27.044670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234a44e7-c0e9-4f13-b2e4-8713fdeb8ac1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:27.046491 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234a44e7-c0e9-4f13-b2e4-8713fdeb8ac1 service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:27.065976 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 550/1111] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:50:28.124521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591353ee-b9e0-4533-a73e-429f292bba09 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:28.125097 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591353ee-b9e0-4533-a73e-429f292bba09 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:28.125097 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591353ee-b9e0-4533-a73e-429f292bba09 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:28.126393 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 562/1112] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 31 14:50:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:28.131378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a528d70-7b39-478a-b0d4-34a526f37f16 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:28.131918 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a528d70-7b39-478a-b0d4-34a526f37f16 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:28.132011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a528d70-7b39-478a-b0d4-34a526f37f16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:28.206114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-8a528d70-7b39-478a-b0d4-34a526f37f16 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:28.207310 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-8a528d70-7b39-478a-b0d4-34a526f37f16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:28.249198 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:28.303449 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 551/1113] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 31 14:50:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:28.455346 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591353ee-b9e0-4533-a73e-429f292bba09 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:28.456901 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-591353ee-b9e0-4533-a73e-429f292bba09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:28.603286 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591353ee-b9e0-4533-a73e-429f292bba09 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:28.658143 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:28.724764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d59aca9-6813-4e2a-94bf-2ed6c18a0473 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:28.725472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d59aca9-6813-4e2a-94bf-2ed6c18a0473 service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:28.725801 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d59aca9-6813-4e2a-94bf-2ed6c18a0473 service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:28.726119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d59aca9-6813-4e2a-94bf-2ed6c18a0473 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:28.727959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2d59aca9-6813-4e2a-94bf-2ed6c18a0473 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:28.828552 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d59aca9-6813-4e2a-94bf-2ed6c18a0473 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:28.830350 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d59aca9-6813-4e2a-94bf-2ed6c18a0473 service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:28.850669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 563/1114] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:50:29.889960 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09049118-afac-4b39-b0ef-e4253d27f735 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:29.890499 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09049118-afac-4b39-b0ef-e4253d27f735 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:29.890499 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09049118-afac-4b39-b0ef-e4253d27f735 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:29.891825 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 552/1115] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 31 14:50:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:29.896792 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6471cf-bd03-402b-9275-d7780d992578 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:29.897199 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6471cf-bd03-402b-9275-d7780d992578 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:29.897364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6471cf-bd03-402b-9275-d7780d992578 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:29.972188 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-7e6471cf-bd03-402b-9275-d7780d992578 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:29.973382 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7e6471cf-bd03-402b-9275-d7780d992578 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:29.975376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:30.072498 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 564/1116] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 31 14:50:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:30.223874 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09049118-afac-4b39-b0ef-e4253d27f735 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:30.225395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-09049118-afac-4b39-b0ef-e4253d27f735 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:30.372278 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09049118-afac-4b39-b0ef-e4253d27f735 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:30.398576 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:30.487132 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809b5c97-e892-4bdd-8919-6da3b13b322b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:30.487686 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809b5c97-e892-4bdd-8919-6da3b13b322b service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:30.487967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809b5c97-e892-4bdd-8919-6da3b13b322b service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:30.488239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809b5c97-e892-4bdd-8919-6da3b13b322b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:30.489843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-809b5c97-e892-4bdd-8919-6da3b13b322b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:30.583036 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809b5c97-e892-4bdd-8919-6da3b13b322b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:30.584671 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809b5c97-e892-4bdd-8919-6da3b13b322b service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:30.603453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 553/1117] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:50:32.677635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8075aad2-f143-4a0a-ab4b-e8d8a0f4bce9 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:32.678166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8075aad2-f143-4a0a-ab4b-e8d8a0f4bce9 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:32.678166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8075aad2-f143-4a0a-ab4b-e8d8a0f4bce9 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:32.679510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 565/1118] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 31 14:50:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:32.685217 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bf413e-665f-4a4a-8a3a-27d79487cf79 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:32.685658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bf413e-665f-4a4a-8a3a-27d79487cf79 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:32.685765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bf413e-665f-4a4a-8a3a-27d79487cf79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:32.727207 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:32.755423 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-e7bf413e-665f-4a4a-8a3a-27d79487cf79 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:32.756677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e7bf413e-665f-4a4a-8a3a-27d79487cf79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:32.853943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 554/1119] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 31 14:50:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:33.142245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8075aad2-f143-4a0a-ab4b-e8d8a0f4bce9 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:33.144360 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8075aad2-f143-4a0a-ab4b-e8d8a0f4bce9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:33.251169 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8075aad2-f143-4a0a-ab4b-e8d8a0f4bce9 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:33.284520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:33.329986 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12b6bb1-2b26-4c4f-b678-32c6422bf5d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:33.330592 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12b6bb1-2b26-4c4f-b678-32c6422bf5d2 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:33.330870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12b6bb1-2b26-4c4f-b678-32c6422bf5d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:33.331140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12b6bb1-2b26-4c4f-b678-32c6422bf5d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:33.332788 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-f12b6bb1-2b26-4c4f-b678-32c6422bf5d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:33.442482 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12b6bb1-2b26-4c4f-b678-32c6422bf5d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:33.444007 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12b6bb1-2b26-4c4f-b678-32c6422bf5d2 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:33.463340 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 566/1120] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:50:35.779290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f471ff63-3576-45b6-a7ed-5022f670f545 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:35.779884 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f471ff63-3576-45b6-a7ed-5022f670f545 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:35.779884 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f471ff63-3576-45b6-a7ed-5022f670f545 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:35.781207 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 555/1121] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 31 14:50:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:35.786463 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c93534-3385-487d-8803-2c337a549a5e None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:35.786858 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c93534-3385-487d-8803-2c337a549a5e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:35.786959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c93534-3385-487d-8803-2c337a549a5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:35.851797 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-e7c93534-3385-487d-8803-2c337a549a5e None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:35.852997 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e7c93534-3385-487d-8803-2c337a549a5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:35.866709 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:35.951910 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 567/1122] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 31 14:50:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:36.239843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f471ff63-3576-45b6-a7ed-5022f670f545 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:36.241527 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-f471ff63-3576-45b6-a7ed-5022f670f545 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:36.344549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f471ff63-3576-45b6-a7ed-5022f670f545 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:36.358222 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:36.420106 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487556e5-46f1-4adc-80fe-c3ede4571f81 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:36.420698 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487556e5-46f1-4adc-80fe-c3ede4571f81 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:36.420970 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487556e5-46f1-4adc-80fe-c3ede4571f81 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:36.421236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487556e5-46f1-4adc-80fe-c3ede4571f81 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:36.422814 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-487556e5-46f1-4adc-80fe-c3ede4571f81 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:36.533842 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487556e5-46f1-4adc-80fe-c3ede4571f81 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:36.534588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487556e5-46f1-4adc-80fe-c3ede4571f81 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:36.552855 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 556/1123] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:50:39.078511 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2e3947-d207-4c01-9b42-dcc0dc4b2a5a None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:39.079047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2e3947-d207-4c01-9b42-dcc0dc4b2a5a None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:39.079047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2e3947-d207-4c01-9b42-dcc0dc4b2a5a None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:39.080431 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 568/1124] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 31 14:50:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:39.086411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c3e9e6-ccc3-4b20-8415-3387ae911702 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:39.086901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c3e9e6-ccc3-4b20-8415-3387ae911702 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:39.087029 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c3e9e6-ccc3-4b20-8415-3387ae911702 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:39.136259 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:39.157620 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-05c3e9e6-ccc3-4b20-8415-3387ae911702 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:39.159037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-05c3e9e6-ccc3-4b20-8415-3387ae911702 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:39.267107 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 557/1125] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 31 14:50:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:39.555410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2e3947-d207-4c01-9b42-dcc0dc4b2a5a None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:39.557282 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0d2e3947-d207-4c01-9b42-dcc0dc4b2a5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:39.670072 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2e3947-d207-4c01-9b42-dcc0dc4b2a5a None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:39.709469 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:39.743470 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0017cce-8836-4edb-a3c5-a5ab30f2dc44 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:39.744095 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0017cce-8836-4edb-a3c5-a5ab30f2dc44 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:39.744373 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0017cce-8836-4edb-a3c5-a5ab30f2dc44 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:39.744645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0017cce-8836-4edb-a3c5-a5ab30f2dc44 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:39.746283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-c0017cce-8836-4edb-a3c5-a5ab30f2dc44 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:39.855931 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0017cce-8836-4edb-a3c5-a5ab30f2dc44 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:39.857283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0017cce-8836-4edb-a3c5-a5ab30f2dc44 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:39.875267 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 569/1126] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:50:42.157404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ae871f-e6b3-43de-9ef5-c4959d5b205a None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:42.157988 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ae871f-e6b3-43de-9ef5-c4959d5b205a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:42.157988 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ae871f-e6b3-43de-9ef5-c4959d5b205a None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:42.159309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 558/1127] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 31 14:50:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:42.164450 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc70c06-1053-4324-8ef8-bae233e9a75e None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:42.164841 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc70c06-1053-4324-8ef8-bae233e9a75e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:42.165026 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc70c06-1053-4324-8ef8-bae233e9a75e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:42.231326 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-8dc70c06-1053-4324-8ef8-bae233e9a75e None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:42.232434 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8dc70c06-1053-4324-8ef8-bae233e9a75e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:42.254009 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:42.332157 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 570/1128] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 31 14:50:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:50:42.617677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ae871f-e6b3-43de-9ef5-c4959d5b205a None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:42.619218 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-40ae871f-e6b3-43de-9ef5-c4959d5b205a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:42.724858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ae871f-e6b3-43de-9ef5-c4959d5b205a None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:42.747907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:42.802353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a7c44a-2ee8-458b-8b3e-67114b4a235c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:42.802879 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a7c44a-2ee8-458b-8b3e-67114b4a235c service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:42.803161 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a7c44a-2ee8-458b-8b3e-67114b4a235c service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:42.803433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a7c44a-2ee8-458b-8b3e-67114b4a235c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:42.805077 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-56a7c44a-2ee8-458b-8b3e-67114b4a235c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:42.915822 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a7c44a-2ee8-458b-8b3e-67114b4a235c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:42.917258 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a7c44a-2ee8-458b-8b3e-67114b4a235c service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:42.935701 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 559/1129] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:50:45.509137 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5035293c-c897-45dd-9272-e7e0d082908a None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:45.509726 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5035293c-c897-45dd-9272-e7e0d082908a None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:45.509726 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5035293c-c897-45dd-9272-e7e0d082908a None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:45.510919 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 571/1130] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 31 14:50:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:45.517049 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90100f1-d82d-487a-bddc-eb5df1514785 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:45.517471 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90100f1-d82d-487a-bddc-eb5df1514785 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:45.517594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90100f1-d82d-487a-bddc-eb5df1514785 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:45.590137 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:45.593724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-d90100f1-d82d-487a-bddc-eb5df1514785 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:45.595275 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d90100f1-d82d-487a-bddc-eb5df1514785 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:45.697046 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 560/1131] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 31 14:50:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:45.981607 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5035293c-c897-45dd-9272-e7e0d082908a None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:45.983147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5035293c-c897-45dd-9272-e7e0d082908a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:46.088758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5035293c-c897-45dd-9272-e7e0d082908a None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:46.154082 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:46.184525 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e9abee-b51c-42c7-b40a-b1a43874ca82 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:46.185134 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e9abee-b51c-42c7-b40a-b1a43874ca82 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:46.185415 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e9abee-b51c-42c7-b40a-b1a43874ca82 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:46.185690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e9abee-b51c-42c7-b40a-b1a43874ca82 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:46.187292 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-09e9abee-b51c-42c7-b40a-b1a43874ca82 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:46.297471 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e9abee-b51c-42c7-b40a-b1a43874ca82 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:46.298815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e9abee-b51c-42c7-b40a-b1a43874ca82 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:46.317140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 572/1132] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:50:48.530208 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e24142-c249-4d26-b7a4-f38690c12a91 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:48.530769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e24142-c249-4d26-b7a4-f38690c12a91 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:48.530769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e24142-c249-4d26-b7a4-f38690c12a91 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:48.532289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 561/1133] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 31 14:50:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:48.537443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7f1fd1-ee58-4438-8f83-4f83651f5860 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:48.537899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7f1fd1-ee58-4438-8f83-4f83651f5860 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:48.537994 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7f1fd1-ee58-4438-8f83-4f83651f5860 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:48.606917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-0d7f1fd1-ee58-4438-8f83-4f83651f5860 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:48.608151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0d7f1fd1-ee58-4438-8f83-4f83651f5860 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:48.636847 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:48.705919 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 573/1134] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 31 14:50:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:49.070009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e24142-c249-4d26-b7a4-f38690c12a91 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:49.071544 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-09e24142-c249-4d26-b7a4-f38690c12a91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:49.174720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e24142-c249-4d26-b7a4-f38690c12a91 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:49.203942 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:49.250400 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a4f79d-fbd4-413c-a4ca-50e1a1bea90f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:49.251000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a4f79d-fbd4-413c-a4ca-50e1a1bea90f service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:49.251278 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a4f79d-fbd4-413c-a4ca-50e1a1bea90f service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:49.251573 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a4f79d-fbd4-413c-a4ca-50e1a1bea90f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:49.253176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a4a4f79d-fbd4-413c-a4ca-50e1a1bea90f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:49.363412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a4f79d-fbd4-413c-a4ca-50e1a1bea90f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:49.364794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a4f79d-fbd4-413c-a4ca-50e1a1bea90f service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:49.383283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 562/1135] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:50:51.680914 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acff7acb-c1b7-44c1-91dd-864cc633b7ce None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:51.681442 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acff7acb-c1b7-44c1-91dd-864cc633b7ce None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:51.681570 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acff7acb-c1b7-44c1-91dd-864cc633b7ce None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:51.682832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 574/1136] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 31 14:50:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:51.688658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86415219-f8ac-46e2-bad9-78d680679eb0 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:51.689065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86415219-f8ac-46e2-bad9-78d680679eb0 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:51.689241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86415219-f8ac-46e2-bad9-78d680679eb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:51.756530 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-86415219-f8ac-46e2-bad9-78d680679eb0 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:51.757753 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-86415219-f8ac-46e2-bad9-78d680679eb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:51.766857 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:51.857305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 563/1137] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 31 14:50:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:52.145029 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acff7acb-c1b7-44c1-91dd-864cc633b7ce None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:52.146573 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-acff7acb-c1b7-44c1-91dd-864cc633b7ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:52.253234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acff7acb-c1b7-44c1-91dd-864cc633b7ce None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:52.311781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:52.331994 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e5d980-6625-4202-9f5c-7e8fe5b04ee7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:52.332598 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e5d980-6625-4202-9f5c-7e8fe5b04ee7 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:52.332879 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e5d980-6625-4202-9f5c-7e8fe5b04ee7 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:52.333153 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e5d980-6625-4202-9f5c-7e8fe5b04ee7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:52.334763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-99e5d980-6625-4202-9f5c-7e8fe5b04ee7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:52.446951 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e5d980-6625-4202-9f5c-7e8fe5b04ee7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:52.448413 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e5d980-6625-4202-9f5c-7e8fe5b04ee7 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:52.469105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 575/1138] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:50:54.683741 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b32916-1be0-4120-a9d4-bfefabf51772 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:54.684356 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b32916-1be0-4120-a9d4-bfefabf51772 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:54.684532 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b32916-1be0-4120-a9d4-bfefabf51772 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:54.686038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 564/1139] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 31 14:50:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:54.691374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a32cce-dcc0-4be8-a33a-ac0563f00495 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:54.691820 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a32cce-dcc0-4be8-a33a-ac0563f00495 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:54.691975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a32cce-dcc0-4be8-a33a-ac0563f00495 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:54.757689 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-90a32cce-dcc0-4be8-a33a-ac0563f00495 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:54.758893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-90a32cce-dcc0-4be8-a33a-ac0563f00495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:54.797818 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:54.856325 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 576/1140] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 31 14:50:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:55.142808 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b32916-1be0-4120-a9d4-bfefabf51772 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:55.144423 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-03b32916-1be0-4120-a9d4-bfefabf51772 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:55.253133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b32916-1be0-4120-a9d4-bfefabf51772 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:55.291289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:55.328729 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0e0d04-9e62-4eee-bb12-5e942ca2b24e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:55.329336 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0e0d04-9e62-4eee-bb12-5e942ca2b24e service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:55.329615 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0e0d04-9e62-4eee-bb12-5e942ca2b24e service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:55.329887 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0e0d04-9e62-4eee-bb12-5e942ca2b24e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:55.331502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ee0e0d04-9e62-4eee-bb12-5e942ca2b24e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:55.439859 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0e0d04-9e62-4eee-bb12-5e942ca2b24e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:55.441301 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0e0d04-9e62-4eee-bb12-5e942ca2b24e service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:55.459152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 565/1141] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:50:57.665878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f3dd18-b135-4a31-89a6-8efc58459007 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:57.666420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f3dd18-b135-4a31-89a6-8efc58459007 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:57.666420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f3dd18-b135-4a31-89a6-8efc58459007 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:57.667851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 577/1142] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 31 14:50:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:50:57.674072 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382257a2-2981-4655-a7a1-6e0bbf32b5e2 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:57.674482 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382257a2-2981-4655-a7a1-6e0bbf32b5e2 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:57.674640 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382257a2-2981-4655-a7a1-6e0bbf32b5e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:57.741645 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-382257a2-2981-4655-a7a1-6e0bbf32b5e2 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:50:57.742854 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-382257a2-2981-4655-a7a1-6e0bbf32b5e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:57.761143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:57.844125 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 566/1143] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 31 14:50:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:50:58.129452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f3dd18-b135-4a31-89a6-8efc58459007 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:50:58.131313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-37f3dd18-b135-4a31-89a6-8efc58459007 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:58.238328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f3dd18-b135-4a31-89a6-8efc58459007 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:50:58.306161 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:50:58.312979 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe30d69-c083-4e36-a7ba-9c27d53e4147 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:50:58.313587 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe30d69-c083-4e36-a7ba-9c27d53e4147 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:50:58.313863 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe30d69-c083-4e36-a7ba-9c27d53e4147 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:50:58.314133 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe30d69-c083-4e36-a7ba-9c27d53e4147 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:50:58.315764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-bfe30d69-c083-4e36-a7ba-9c27d53e4147 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:50:58.426017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe30d69-c083-4e36-a7ba-9c27d53e4147 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:50:58.427460 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe30d69-c083-4e36-a7ba-9c27d53e4147 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:50:58.445461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 578/1144] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:50:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:51:00.630644 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c226043-597d-489d-94d5-02939652cb4d None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:00.631167 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c226043-597d-489d-94d5-02939652cb4d None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:00.631167 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c226043-597d-489d-94d5-02939652cb4d None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:00.632614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 567/1145] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 31 14:51:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:51:00.637912 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cbcb83-82c2-4957-920c-33f15ca87cf7 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:00.638340 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cbcb83-82c2-4957-920c-33f15ca87cf7 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:00.638497 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cbcb83-82c2-4957-920c-33f15ca87cf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:00.703151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-12cbcb83-82c2-4957-920c-33f15ca87cf7 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:51:00.704388 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-12cbcb83-82c2-4957-920c-33f15ca87cf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:00.756293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:00.806406 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 579/1146] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 31 14:51:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:51:01.090892 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c226043-597d-489d-94d5-02939652cb4d None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:51:01.092608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-1c226043-597d-489d-94d5-02939652cb4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:01.196547 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c226043-597d-489d-94d5-02939652cb4d None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:51:01.245754 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:01.273228 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeabf3e7-1dc5-4d15-8813-15340e87fbbb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:51:01.273830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeabf3e7-1dc5-4d15-8813-15340e87fbbb service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:01.274112 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeabf3e7-1dc5-4d15-8813-15340e87fbbb service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:01.274385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeabf3e7-1dc5-4d15-8813-15340e87fbbb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:01.276021 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-aeabf3e7-1dc5-4d15-8813-15340e87fbbb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:01.386400 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeabf3e7-1dc5-4d15-8813-15340e87fbbb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:51:01.387948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeabf3e7-1dc5-4d15-8813-15340e87fbbb service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:51:01.405890 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 568/1147] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:51:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:51:10.805003 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=22 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:11.403615 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=66 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:21.748657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9a957f-a8f2-475d-b06b-94bb179f44a2 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:21.749216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9a957f-a8f2-475d-b06b-94bb179f44a2 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:21.749216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9a957f-a8f2-475d-b06b-94bb179f44a2 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:21.750575 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 580/1148] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:51:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:51:21.756180 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742b010b-d181-45a9-ac2e-cac01f6fa5f9 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:21.756620 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742b010b-d181-45a9-ac2e-cac01f6fa5f9 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:21.756796 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742b010b-d181-45a9-ac2e-cac01f6fa5f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:21.824819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-742b010b-d181-45a9-ac2e-cac01f6fa5f9 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:51:21.825958 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-742b010b-d181-45a9-ac2e-cac01f6fa5f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:21.925657 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 569/1149] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:51:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:51:22.234804 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b9dca1-978f-4a07-b568-65f113e7aa9d None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:22.235089 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b9dca1-978f-4a07-b568-65f113e7aa9d None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:22.235394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b9dca1-978f-4a07-b568-65f113e7aa9d None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:22.236751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 581/1150] 10.222.0.36 () {54 vars in 962 bytes} [Mon Oct 31 14:51:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:51:22.242596 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a2ab73-55da-4c7f-b4f3-78f63237d350 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:22.243002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a2ab73-55da-4c7f-b4f3-78f63237d350 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:22.243276 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a2ab73-55da-4c7f-b4f3-78f63237d350 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:22.319289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-68a2ab73-55da-4c7f-b4f3-78f63237d350 None None] MFA Rules not processed for user `a9808dcba0b64027bb70e31d1569957d`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:51:22.320536 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-68a2ab73-55da-4c7f-b4f3-78f63237d350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:22.341099 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:22.396348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 570/1151] 10.222.0.36 () {58 vars in 1059 bytes} [Mon Oct 31 14:51:22 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:51:22.404216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b9dca1-978f-4a07-b568-65f113e7aa9d None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:51:22.405890 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-53b9dca1-978f-4a07-b568-65f113e7aa9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:22.510385 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b9dca1-978f-4a07-b568-65f113e7aa9d None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:51:22.583515 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1364cc3a-1782-4069-8ac2-2bceeb710361 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9808dcba0b64027bb70e31d1569957d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:51:22.584144 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1364cc3a-1782-4069-8ac2-2bceeb710361 service glance] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:22.584417 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1364cc3a-1782-4069-8ac2-2bceeb710361 service glance] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:22.584687 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1364cc3a-1782-4069-8ac2-2bceeb710361 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:22.586268 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1364cc3a-1782-4069-8ac2-2bceeb710361 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:22.641031 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:22.697828 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1364cc3a-1782-4069-8ac2-2bceeb710361 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:51:22.699795 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1364cc3a-1782-4069-8ac2-2bceeb710361 service glance] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:51:22.720444 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 582/1152] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:51:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:51:25.014736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed68cdde-1ee2-4d0d-b17a-0100fa75ea91 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:25.015294 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed68cdde-1ee2-4d0d-b17a-0100fa75ea91 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:25.015294 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed68cdde-1ee2-4d0d-b17a-0100fa75ea91 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:25.016703 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 571/1153] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:51:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:51:25.021945 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc57f904-cba6-46e3-82ec-bde0793d0fdc None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:25.022404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc57f904-cba6-46e3-82ec-bde0793d0fdc None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:25.022510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc57f904-cba6-46e3-82ec-bde0793d0fdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:25.087771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-cc57f904-cba6-46e3-82ec-bde0793d0fdc None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:51:25.088964 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-cc57f904-cba6-46e3-82ec-bde0793d0fdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:25.184692 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 583/1154] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:51:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:51:25.463513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed68cdde-1ee2-4d0d-b17a-0100fa75ea91 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:51:25.465539 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ed68cdde-1ee2-4d0d-b17a-0100fa75ea91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:25.591630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed68cdde-1ee2-4d0d-b17a-0100fa75ea91 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:51:25.641674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:25.685113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f284ccf0-5229-4c2d-8ec2-c7f7f010bdd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:51:25.685715 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f284ccf0-5229-4c2d-8ec2-c7f7f010bdd8 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:25.685995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f284ccf0-5229-4c2d-8ec2-c7f7f010bdd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:25.686267 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f284ccf0-5229-4c2d-8ec2-c7f7f010bdd8 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:25.686905 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f284ccf0-5229-4c2d-8ec2-c7f7f010bdd8 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:51:25.688047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: WARNING py.warnings [None req-f284ccf0-5229-4c2d-8ec2-c7f7f010bdd8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 31 14:51:25.688047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: warnings.warn(msg) Oct 31 14:51:25.688047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: Oct 31 14:51:25.689193 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f284ccf0-5229-4c2d-8ec2-c7f7f010bdd8 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:51:25.709209 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 572/1155] 10.222.0.36 () {56 vars in 1108 bytes} [Mon Oct 31 14:51:25 2022] GET /identity/v3/projects => generated 2553 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:51:27.884866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cce2cc-9c9c-47b1-a96d-555506e0ec1d None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:27.885394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cce2cc-9c9c-47b1-a96d-555506e0ec1d None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:27.885394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cce2cc-9c9c-47b1-a96d-555506e0ec1d None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:27.887199 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 584/1156] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:51:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:51:27.892905 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8588f74-18bb-4750-af66-077891796153 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:27.893374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8588f74-18bb-4750-af66-077891796153 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:27.893469 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8588f74-18bb-4750-af66-077891796153 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:27.958369 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-c8588f74-18bb-4750-af66-077891796153 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:51:27.959604 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-c8588f74-18bb-4750-af66-077891796153 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:28.054563 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 573/1157] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:51:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:51:28.346581 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cce2cc-9c9c-47b1-a96d-555506e0ec1d None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:51:28.348397 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-71cce2cc-9c9c-47b1-a96d-555506e0ec1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:28.358082 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:28.491474 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cce2cc-9c9c-47b1-a96d-555506e0ec1d None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:51:28.571627 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:28.604966 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95bfa86-42ba-40c8-8865-9bea69f3ea13 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:51:28.606014 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95bfa86-42ba-40c8-8865-9bea69f3ea13 service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:28.606293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95bfa86-42ba-40c8-8865-9bea69f3ea13 service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:28.606562 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95bfa86-42ba-40c8-8865-9bea69f3ea13 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:28.608198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-f95bfa86-42ba-40c8-8865-9bea69f3ea13 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:28.698071 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95bfa86-42ba-40c8-8865-9bea69f3ea13 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:51:28.699969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95bfa86-42ba-40c8-8865-9bea69f3ea13 service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:51:28.718241 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 585/1158] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:51:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:51:31.080075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b691987f-2326-4531-a5a8-8762c306ba63 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:31.080633 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b691987f-2326-4531-a5a8-8762c306ba63 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:31.080633 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b691987f-2326-4531-a5a8-8762c306ba63 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:31.081957 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 574/1159] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:51:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:51:31.087287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da552beb-ee6c-4da1-9965-7e25afa0e643 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:31.087760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da552beb-ee6c-4da1-9965-7e25afa0e643 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:31.087870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da552beb-ee6c-4da1-9965-7e25afa0e643 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:31.154656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-da552beb-ee6c-4da1-9965-7e25afa0e643 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:51:31.155924 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-da552beb-ee6c-4da1-9965-7e25afa0e643 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:31.201883 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:31.252466 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 586/1160] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:51:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:51:31.541062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b691987f-2326-4531-a5a8-8762c306ba63 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:51:31.543222 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-b691987f-2326-4531-a5a8-8762c306ba63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:31.568242 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:31.648063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b691987f-2326-4531-a5a8-8762c306ba63 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:51:31.722691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ec003b-0f7e-4f78-bdc3-a0cf2dddb190 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:51:31.723292 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ec003b-0f7e-4f78-bdc3-a0cf2dddb190 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:31.723590 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ec003b-0f7e-4f78-bdc3-a0cf2dddb190 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:31.723866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ec003b-0f7e-4f78-bdc3-a0cf2dddb190 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:31.725461 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-c1ec003b-0f7e-4f78-bdc3-a0cf2dddb190 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:31.798888 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:31.833459 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ec003b-0f7e-4f78-bdc3-a0cf2dddb190 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:51:31.834905 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ec003b-0f7e-4f78-bdc3-a0cf2dddb190 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:51:31.855028 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 575/1161] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:51:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:51:34.076580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe9da02-90c3-4f67-b89a-b1c3af64a684 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:34.077152 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe9da02-90c3-4f67-b89a-b1c3af64a684 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:34.077152 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe9da02-90c3-4f67-b89a-b1c3af64a684 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:34.078459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 587/1162] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:51:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:51:34.083687 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf2d280-3cfe-4294-8db2-f419c0a813c1 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:34.084131 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf2d280-3cfe-4294-8db2-f419c0a813c1 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:34.084237 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf2d280-3cfe-4294-8db2-f419c0a813c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:34.150360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-bdf2d280-3cfe-4294-8db2-f419c0a813c1 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:51:34.151610 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-bdf2d280-3cfe-4294-8db2-f419c0a813c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:34.250171 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 576/1163] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:51:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:51:34.534135 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe9da02-90c3-4f67-b89a-b1c3af64a684 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:51:34.535716 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ebe9da02-90c3-4f67-b89a-b1c3af64a684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:34.640164 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe9da02-90c3-4f67-b89a-b1c3af64a684 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:51:34.710981 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:34.712588 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12acf0a-599f-468b-b58f-396aca282d29 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:51:34.713300 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12acf0a-599f-468b-b58f-396aca282d29 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:34.713587 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12acf0a-599f-468b-b58f-396aca282d29 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:34.714058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12acf0a-599f-468b-b58f-396aca282d29 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:34.716507 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a12acf0a-599f-468b-b58f-396aca282d29 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:34.828114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12acf0a-599f-468b-b58f-396aca282d29 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:51:34.829445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12acf0a-599f-468b-b58f-396aca282d29 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:51:34.847263 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 588/1164] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:51:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:51:37.115572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3e142b-fa24-4595-b261-bf5349ebc2d4 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:37.116146 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3e142b-fa24-4595-b261-bf5349ebc2d4 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:37.116146 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3e142b-fa24-4595-b261-bf5349ebc2d4 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:37.117545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 577/1165] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:51:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:51:37.123099 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7843fd9-9a33-4021-9ef8-720fc553dfbc None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:37.123534 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7843fd9-9a33-4021-9ef8-720fc553dfbc None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:37.123691 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7843fd9-9a33-4021-9ef8-720fc553dfbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:37.189548 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-e7843fd9-9a33-4021-9ef8-720fc553dfbc None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:51:37.190749 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e7843fd9-9a33-4021-9ef8-720fc553dfbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:37.243983 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:37.288671 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 589/1166] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:51:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:51:37.582052 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3e142b-fa24-4595-b261-bf5349ebc2d4 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:51:37.583635 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2b3e142b-fa24-4595-b261-bf5349ebc2d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:37.619990 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:37.686369 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3e142b-fa24-4595-b261-bf5349ebc2d4 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:51:37.759936 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba0bc1e-7e37-48c0-9c60-1d14c1004450 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:51:37.760536 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba0bc1e-7e37-48c0-9c60-1d14c1004450 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:37.760815 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba0bc1e-7e37-48c0-9c60-1d14c1004450 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:37.761592 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba0bc1e-7e37-48c0-9c60-1d14c1004450 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:37.763429 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-dba0bc1e-7e37-48c0-9c60-1d14c1004450 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:37.843710 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:37.873075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba0bc1e-7e37-48c0-9c60-1d14c1004450 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:51:37.874513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba0bc1e-7e37-48c0-9c60-1d14c1004450 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:51:37.892813 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 578/1167] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:51:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:51:40.276293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7248f1-a628-4191-a217-dc497e5c62af None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:40.277361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7248f1-a628-4191-a217-dc497e5c62af None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:40.277361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7248f1-a628-4191-a217-dc497e5c62af None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:40.278715 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 590/1168] 10.222.0.36 () {54 vars in 927 bytes} [Mon Oct 31 14:51:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:51:40.284061 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0505b8-226a-4459-bc6f-a5fc80a41ba4 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:40.284501 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0505b8-226a-4459-bc6f-a5fc80a41ba4 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:40.284622 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0505b8-226a-4459-bc6f-a5fc80a41ba4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:40.353201 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-fd0505b8-226a-4459-bc6f-a5fc80a41ba4 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:51:40.354444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-fd0505b8-226a-4459-bc6f-a5fc80a41ba4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:40.453450 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 579/1169] 10.222.0.36 () {58 vars in 1024 bytes} [Mon Oct 31 14:51:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:51:40.744870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7248f1-a628-4191-a217-dc497e5c62af None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:51:40.746555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-3a7248f1-a628-4191-a217-dc497e5c62af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:40.848929 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7248f1-a628-4191-a217-dc497e5c62af None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:51:40.922851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e982dbc0-3e6f-4a2a-907c-14be9efafe1c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:51:40.923459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e982dbc0-3e6f-4a2a-907c-14be9efafe1c service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:51:40.923763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e982dbc0-3e6f-4a2a-907c-14be9efafe1c service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:51:40.924032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e982dbc0-3e6f-4a2a-907c-14be9efafe1c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:51:40.925654 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e982dbc0-3e6f-4a2a-907c-14be9efafe1c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:51:40.934131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:41.034694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e982dbc0-3e6f-4a2a-907c-14be9efafe1c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:51:41.036169 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e982dbc0-3e6f-4a2a-907c-14be9efafe1c service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:51:41.053951 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 591/1170] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:51:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:51:50.451952 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=82 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:51:51.051797 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=50 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:53:02.559975 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7182cd6a-4100-4916-8d27-8208c85a1b49 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:53:02.560470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7182cd6a-4100-4916-8d27-8208c85a1b49 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:53:02.560562 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7182cd6a-4100-4916-8d27-8208c85a1b49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:53:02.638884 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-7182cd6a-4100-4916-8d27-8208c85a1b49 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:53:02.640278 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-7182cd6a-4100-4916-8d27-8208c85a1b49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:53:02.738411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 580/1171] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:53:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:53:02.746448 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3afbe5a-90e5-453c-be17-b0e3ed04341e None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:53:02.746928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3afbe5a-90e5-453c-be17-b0e3ed04341e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:53:02.747036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3afbe5a-90e5-453c-be17-b0e3ed04341e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:53:02.824696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-a3afbe5a-90e5-453c-be17-b0e3ed04341e None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:53:02.826245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a3afbe5a-90e5-453c-be17-b0e3ed04341e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:53:02.922674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 592/1172] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:53:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:53:02.927293 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7182cd6a-4100-4916-8d27-8208c85a1b49 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:53:02.928562 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-7182cd6a-4100-4916-8d27-8208c85a1b49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:53:02.940800 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7182cd6a-4100-4916-8d27-8208c85a1b49 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:53:03.003243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:53:03.037634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f663033-72e0-4a07-b43c-79e296b1aad6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:53:03.038237 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f663033-72e0-4a07-b43c-79e296b1aad6 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:53:03.038517 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f663033-72e0-4a07-b43c-79e296b1aad6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:53:03.038788 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f663033-72e0-4a07-b43c-79e296b1aad6 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:53:03.039428 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f663033-72e0-4a07-b43c-79e296b1aad6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:53:03.039767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f663033-72e0-4a07-b43c-79e296b1aad6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:53:03.040868 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: WARNING py.warnings [None req-8f663033-72e0-4a07-b43c-79e296b1aad6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 31 14:53:03.040868 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: warnings.warn(msg) Oct 31 14:53:03.040868 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: Oct 31 14:53:03.041780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f663033-72e0-4a07-b43c-79e296b1aad6 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:53:03.052833 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 581/1173] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:53:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:53:03.062778 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3afbe5a-90e5-453c-be17-b0e3ed04341e None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:53:03.072432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3afbe5a-90e5-453c-be17-b0e3ed04341e None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:53:03.133499 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:53:03.168452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cecf89-1fec-41df-8977-3c4107adab1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:53:03.169058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cecf89-1fec-41df-8977-3c4107adab1c admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:53:03.169340 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cecf89-1fec-41df-8977-3c4107adab1c admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:53:03.169608 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cecf89-1fec-41df-8977-3c4107adab1c admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:53:03.170234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84cecf89-1fec-41df-8977-3c4107adab1c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:53:03.171246 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: WARNING py.warnings [None req-84cecf89-1fec-41df-8977-3c4107adab1c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 31 14:53:03.171246 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: warnings.warn(msg) Oct 31 14:53:03.171246 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: Oct 31 14:53:03.172466 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84cecf89-1fec-41df-8977-3c4107adab1c admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:53:03.227541 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 593/1174] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:53:03 2022] POST /identity/v3/projects => generated 377 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:53:03.232471 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f663033-72e0-4a07-b43c-79e296b1aad6 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:53:03.234024 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-8f663033-72e0-4a07-b43c-79e296b1aad6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:53:03.267682 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f663033-72e0-4a07-b43c-79e296b1aad6 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:53:03.362536 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7800ba-ae41-44fd-a3f7-9eda2789bbab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:53:03.363061 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7800ba-ae41-44fd-a3f7-9eda2789bbab admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:53:03.363341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7800ba-ae41-44fd-a3f7-9eda2789bbab admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:53:03.363656 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7800ba-ae41-44fd-a3f7-9eda2789bbab admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:53:03.364291 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7800ba-ae41-44fd-a3f7-9eda2789bbab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:53:03.365301 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: WARNING py.warnings [None req-5c7800ba-ae41-44fd-a3f7-9eda2789bbab admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 31 14:53:03.365301 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: warnings.warn(msg) Oct 31 14:53:03.365301 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: Oct 31 14:53:03.366507 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7800ba-ae41-44fd-a3f7-9eda2789bbab admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:53:03.409124 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 582/1175] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:53:03 2022] POST /identity/v3/users => generated 289 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:53:03.413238 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cecf89-1fec-41df-8977-3c4107adab1c admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:53:03.414811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-84cecf89-1fec-41df-8977-3c4107adab1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:53:03.477314 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cecf89-1fec-41df-8977-3c4107adab1c admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:53:03.561445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:53:03.572787 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ed9a61-7bc0-4fee-944f-d090a9c18f85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:53:03.573388 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ed9a61-7bc0-4fee-944f-d090a9c18f85 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:53:03.573668 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ed9a61-7bc0-4fee-944f-d090a9c18f85 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:53:03.573936 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ed9a61-7bc0-4fee-944f-d090a9c18f85 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:53:03.574577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ed9a61-7bc0-4fee-944f-d090a9c18f85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:53:03.576184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ed9a61-7bc0-4fee-944f-d090a9c18f85 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:53:03.589005 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 594/1176] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:53:03 2022] GET /identity/v3/roles => generated 1420 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:53:03.593233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7800ba-ae41-44fd-a3f7-9eda2789bbab admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:53:03.594744 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5c7800ba-ae41-44fd-a3f7-9eda2789bbab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:53:03.606901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7800ba-ae41-44fd-a3f7-9eda2789bbab admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:53:03.632057 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:53:03.703240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615cc2ff-449f-45fa-8c92-1cf6fedc0dff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:53:03.703874 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615cc2ff-449f-45fa-8c92-1cf6fedc0dff admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:53:03.704153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615cc2ff-449f-45fa-8c92-1cf6fedc0dff admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:53:03.704423 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615cc2ff-449f-45fa-8c92-1cf6fedc0dff admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:53:03.705061 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615cc2ff-449f-45fa-8c92-1cf6fedc0dff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:53:03.706650 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615cc2ff-449f-45fa-8c92-1cf6fedc0dff admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:53:03.719570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 583/1177] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:53:03 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:53:03.723741 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ed9a61-7bc0-4fee-944f-d090a9c18f85 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:53:03.725370 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-16ed9a61-7bc0-4fee-944f-d090a9c18f85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:53:03.760897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ed9a61-7bc0-4fee-944f-d090a9c18f85 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:53:03.856180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105e5ba8-070e-4565-bc56-cb931fd00656 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:53:03.856830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105e5ba8-070e-4565-bc56-cb931fd00656 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:53:03.857106 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105e5ba8-070e-4565-bc56-cb931fd00656 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:53:03.857375 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105e5ba8-070e-4565-bc56-cb931fd00656 admin admin] PATH_INFO: `/v3/projects/451f1a67549f446198f8c57156142f58/users/bd31e41fd6ac4f84944316be81bfdef9/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:53:03.880048 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105e5ba8-070e-4565-bc56-cb931fd00656 admin admin] RBAC: Authorizing `identity:create_grant(project_id=451f1a67549f446198f8c57156142f58, user_id=bd31e41fd6ac4f84944316be81bfdef9, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:53:03.882731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105e5ba8-070e-4565-bc56-cb931fd00656 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:53:03.906608 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 595/1178] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:53:03 2022] PUT /identity/v3/projects/451f1a67549f446198f8c57156142f58/users/bd31e41fd6ac4f84944316be81bfdef9/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:53:03.918814 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b288d79-8aec-4019-95c2-10565c5f36ff None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:53:03.919133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b288d79-8aec-4019-95c2-10565c5f36ff None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:53:03.919415 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b288d79-8aec-4019-95c2-10565c5f36ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:53:03.976144 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-0b288d79-8aec-4019-95c2-10565c5f36ff None None] MFA Rules not processed for user `bd31e41fd6ac4f84944316be81bfdef9`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:53:03.977369 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-0b288d79-8aec-4019-95c2-10565c5f36ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:53:04.055181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 584/1179] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:53:03 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:53:04.068703 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ff62b5-18cf-4cb5-bef4-82571c9beddf None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:53:04.069069 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ff62b5-18cf-4cb5-bef4-82571c9beddf None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:53:04.069349 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ff62b5-18cf-4cb5-bef4-82571c9beddf None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:53:04.070684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 596/1180] 10.222.0.36 () {54 vars in 963 bytes} [Mon Oct 31 14:53:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:53:04.075330 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb12b69f-3f6d-4f43-97f3-6b781ffc6205 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:53:04.075653 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb12b69f-3f6d-4f43-97f3-6b781ffc6205 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:53:04.075928 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb12b69f-3f6d-4f43-97f3-6b781ffc6205 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:53:04.107942 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:53:04.154678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-cb12b69f-3f6d-4f43-97f3-6b781ffc6205 None None] MFA Rules not processed for user `993d553a46044c18a4eec6e408c0969a`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:53:04.155932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-cb12b69f-3f6d-4f43-97f3-6b781ffc6205 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:53:04.232194 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 585/1181] 10.222.0.36 () {58 vars in 1060 bytes} [Mon Oct 31 14:53:04 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:53:04.240001 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ff62b5-18cf-4cb5-bef4-82571c9beddf None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:53:04.241493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-75ff62b5-18cf-4cb5-bef4-82571c9beddf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:53:04.291124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:53:04.345101 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ff62b5-18cf-4cb5-bef4-82571c9beddf None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:53:04.416915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573017a0-2e5d-455d-9cdb-0aba32f0da96 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:53:04.417451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573017a0-2e5d-455d-9cdb-0aba32f0da96 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:53:04.417729 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573017a0-2e5d-455d-9cdb-0aba32f0da96 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:53:04.418002 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573017a0-2e5d-455d-9cdb-0aba32f0da96 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:53:04.419651 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-573017a0-2e5d-455d-9cdb-0aba32f0da96 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:53:04.517103 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573017a0-2e5d-455d-9cdb-0aba32f0da96 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:53:04.518550 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573017a0-2e5d-455d-9cdb-0aba32f0da96 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:53:04.522113 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:53:04.536987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 597/1182] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:53:04 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:53:04.582548 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c648392-e968-4f04-8724-25cc12a65004 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:53:04.582852 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c648392-e968-4f04-8724-25cc12a65004 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:53:04.583122 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c648392-e968-4f04-8724-25cc12a65004 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:53:04.584573 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 586/1183] 10.222.0.36 () {52 vars in 830 bytes} [Mon Oct 31 14:53:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:53:04.588235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573017a0-2e5d-455d-9cdb-0aba32f0da96 service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:53:04.589728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-573017a0-2e5d-455d-9cdb-0aba32f0da96 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:53:04.688018 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573017a0-2e5d-455d-9cdb-0aba32f0da96 service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:53:04.779346 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798001b8-576d-4556-8e7a-362b7b9823fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:53:04.780053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798001b8-576d-4556-8e7a-362b7b9823fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:53:04.780264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798001b8-576d-4556-8e7a-362b7b9823fd admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:53:04.780536 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798001b8-576d-4556-8e7a-362b7b9823fd admin admin] PATH_INFO: `/v3/users/bd31e41fd6ac4f84944316be81bfdef9` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:53:04.793757 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798001b8-576d-4556-8e7a-362b7b9823fd admin admin] RBAC: Authorizing `identity:delete_user(user_id=bd31e41fd6ac4f84944316be81bfdef9)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:53:04.794766 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: WARNING py.warnings [None req-798001b8-576d-4556-8e7a-362b7b9823fd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 31 14:53:04.794766 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: warnings.warn(msg) Oct 31 14:53:04.794766 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: Oct 31 14:53:04.796019 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798001b8-576d-4556-8e7a-362b7b9823fd admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:53:04.805510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:53:04.864035 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-798001b8-576d-4556-8e7a-362b7b9823fd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bd31e41fd6ac4f84944316be81bfdef9'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:53:04.864553 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-798001b8-576d-4556-8e7a-362b7b9823fd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bd31e41fd6ac4f84944316be81bfdef9'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:53:04.892433 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-798001b8-576d-4556-8e7a-362b7b9823fd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bd31e41fd6ac4f84944316be81bfdef9'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:53:04.909958 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-798001b8-576d-4556-8e7a-362b7b9823fd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bd31e41fd6ac4f84944316be81bfdef9'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:53:04.920278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-798001b8-576d-4556-8e7a-362b7b9823fd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bd31e41fd6ac4f84944316be81bfdef9'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:53:04.926930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 598/1184] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:53:04 2022] DELETE /identity/v3/users/bd31e41fd6ac4f84944316be81bfdef9 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:53:04.931719 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a548f00-6385-486c-a0de-4555366227b9 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:53:04.932255 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a548f00-6385-486c-a0de-4555366227b9 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:53:04.932363 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a548f00-6385-486c-a0de-4555366227b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:53:04.997762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-1a548f00-6385-486c-a0de-4555366227b9 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:53:04.998997 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-1a548f00-6385-486c-a0de-4555366227b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:53:05.075703 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:53:05.088978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 587/1185] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:53:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:53:05.099027 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798001b8-576d-4556-8e7a-362b7b9823fd admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:53:05.100320 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-798001b8-576d-4556-8e7a-362b7b9823fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:53:05.200633 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798001b8-576d-4556-8e7a-362b7b9823fd admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:53:05.254865 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:53:05.276025 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4ed661-814c-4a0c-a9d4-02ecd44eccde service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:53:05.276634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4ed661-814c-4a0c-a9d4-02ecd44eccde service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:53:05.276914 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4ed661-814c-4a0c-a9d4-02ecd44eccde service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:53:05.277185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4ed661-814c-4a0c-a9d4-02ecd44eccde service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:53:05.278796 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-fb4ed661-814c-4a0c-a9d4-02ecd44eccde service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:53:05.390865 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4ed661-814c-4a0c-a9d4-02ecd44eccde service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:53:05.392339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4ed661-814c-4a0c-a9d4-02ecd44eccde service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:53:05.410285 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 599/1186] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:53:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:53:05.739308 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a548f00-6385-486c-a0de-4555366227b9 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:53:05.740581 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-1a548f00-6385-486c-a0de-4555366227b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:53:05.752549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a548f00-6385-486c-a0de-4555366227b9 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:53:05.855335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554fb1ce-5bdb-4a9c-94ae-b2da1beb6af8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:53:05.855986 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554fb1ce-5bdb-4a9c-94ae-b2da1beb6af8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:53:05.856266 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554fb1ce-5bdb-4a9c-94ae-b2da1beb6af8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:53:05.856536 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554fb1ce-5bdb-4a9c-94ae-b2da1beb6af8 admin admin] PATH_INFO: `/v3/projects/451f1a67549f446198f8c57156142f58` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:53:05.866368 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554fb1ce-5bdb-4a9c-94ae-b2da1beb6af8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=451f1a67549f446198f8c57156142f58)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:53:05.867388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: WARNING py.warnings [None req-554fb1ce-5bdb-4a9c-94ae-b2da1beb6af8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 31 14:53:05.867388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: warnings.warn(msg) Oct 31 14:53:05.867388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: Oct 31 14:53:05.868623 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554fb1ce-5bdb-4a9c-94ae-b2da1beb6af8 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:53:05.926374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-554fb1ce-5bdb-4a9c-94ae-b2da1beb6af8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '451f1a67549f446198f8c57156142f58', 'request_id': 'req-554fb1ce-5bdb-4a9c-94ae-b2da1beb6af8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:53:05.936943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-554fb1ce-5bdb-4a9c-94ae-b2da1beb6af8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '451f1a67549f446198f8c57156142f58', 'request_id': 'req-554fb1ce-5bdb-4a9c-94ae-b2da1beb6af8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:53:05.942976 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-554fb1ce-5bdb-4a9c-94ae-b2da1beb6af8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '451f1a67549f446198f8c57156142f58', 'request_id': 'req-554fb1ce-5bdb-4a9c-94ae-b2da1beb6af8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:53:05.974617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-554fb1ce-5bdb-4a9c-94ae-b2da1beb6af8 admin admin] The token cache is being invalidate because project 451f1a67549f446198f8c57156142f58 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:53:05.974907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-554fb1ce-5bdb-4a9c-94ae-b2da1beb6af8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:53:05.975433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-554fb1ce-5bdb-4a9c-94ae-b2da1beb6af8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:53:05.976802 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 588/1187] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:53:05 2022] DELETE /identity/v3/projects/451f1a67549f446198f8c57156142f58 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:53:15.408098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=62 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:53:15.972545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:54:31.462512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c04f92-5273-45de-86ca-f1f6a9824475 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:31.463318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c04f92-5273-45de-86ca-f1f6a9824475 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:31.463318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c04f92-5273-45de-86ca-f1f6a9824475 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:31.464814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 600/1188] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 31 14:54:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:54:31.469856 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19f0902-4542-4305-a3df-a2c54bc6bd56 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:31.470302 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19f0902-4542-4305-a3df-a2c54bc6bd56 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:31.470405 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19f0902-4542-4305-a3df-a2c54bc6bd56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:31.551879 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-e19f0902-4542-4305-a3df-a2c54bc6bd56 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:54:31.553284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e19f0902-4542-4305-a3df-a2c54bc6bd56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:54:31.658069 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 589/1189] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 31 14:54:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:54:31.811367 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c04f92-5273-45de-86ca-f1f6a9824475 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:54:31.813493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b3c04f92-5273-45de-86ca-f1f6a9824475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:54:31.929874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c04f92-5273-45de-86ca-f1f6a9824475 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:54:31.935241 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10327d23-e472-475e-9140-ecfebd2eecbb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:54:31.935869 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10327d23-e472-475e-9140-ecfebd2eecbb service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:31.936149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10327d23-e472-475e-9140-ecfebd2eecbb service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:31.936421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10327d23-e472-475e-9140-ecfebd2eecbb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:31.945131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10327d23-e472-475e-9140-ecfebd2eecbb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:54:31.946709 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10327d23-e472-475e-9140-ecfebd2eecbb service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:54:31.955949 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 601/1190] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:54:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:54:33.033393 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a8de70-25cc-48ea-ae54-ec402c8ea05e None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:33.033928 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a8de70-25cc-48ea-ae54-ec402c8ea05e None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:33.033928 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a8de70-25cc-48ea-ae54-ec402c8ea05e None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:33.035271 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 590/1191] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 31 14:54:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:54:33.040269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a104ce3b-6ea4-4a1c-aa90-e0adbe37ba2c None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:33.040667 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a104ce3b-6ea4-4a1c-aa90-e0adbe37ba2c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:33.040816 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a104ce3b-6ea4-4a1c-aa90-e0adbe37ba2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:33.061280 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-a104ce3b-6ea4-4a1c-aa90-e0adbe37ba2c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:54:33.062473 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a104ce3b-6ea4-4a1c-aa90-e0adbe37ba2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:54:33.071059 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 602/1192] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 31 14:54:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:54:33.223378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a8de70-25cc-48ea-ae54-ec402c8ea05e None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:54:33.238619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a8de70-25cc-48ea-ae54-ec402c8ea05e None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:54:33.245066 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fab330-258a-4675-9781-610e58a1b0ce service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:54:33.245684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fab330-258a-4675-9781-610e58a1b0ce service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:33.246008 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fab330-258a-4675-9781-610e58a1b0ce service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:33.246283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fab330-258a-4675-9781-610e58a1b0ce service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:33.254679 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fab330-258a-4675-9781-610e58a1b0ce service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:54:33.256401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fab330-258a-4675-9781-610e58a1b0ce service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:54:33.266071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 591/1193] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:54:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:54:33.451952 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870962d2-ce18-44c4-b8cc-8be8b9ffa3a9 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:33.452169 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870962d2-ce18-44c4-b8cc-8be8b9ffa3a9 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:33.452423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870962d2-ce18-44c4-b8cc-8be8b9ffa3a9 None None] PATH_INFO: `/v3` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:33.453710 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 603/1194] 10.222.0.44 () {54 vars in 925 bytes} [Mon Oct 31 14:54:33 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:54:33.461350 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dddde1f-12e5-4d5a-aa9f-36898bd6abf6 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:33.461637 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dddde1f-12e5-4d5a-aa9f-36898bd6abf6 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:33.461908 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dddde1f-12e5-4d5a-aa9f-36898bd6abf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:33.522111 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-3dddde1f-12e5-4d5a-aa9f-36898bd6abf6 None None] MFA Rules not processed for user `70ea2aaf529b46dabb70b71f4dcc3860`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:54:33.523330 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-3dddde1f-12e5-4d5a-aa9f-36898bd6abf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:54:33.540720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:54:33.595510 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 592/1195] 10.222.0.44 () {58 vars in 1015 bytes} [Mon Oct 31 14:54:33 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:54:37.126234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ba5348-1dd6-4482-9dca-28fb617ac610 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:37.126713 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ba5348-1dd6-4482-9dca-28fb617ac610 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:37.126812 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ba5348-1dd6-4482-9dca-28fb617ac610 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:37.128110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 604/1196] 10.222.0.36 () {54 vars in 961 bytes} [Mon Oct 31 14:54:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:54:37.133297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e09da0-16ef-4930-9631-d4e00b9dbac7 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:37.133708 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e09da0-16ef-4930-9631-d4e00b9dbac7 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:37.133840 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e09da0-16ef-4930-9631-d4e00b9dbac7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:37.152892 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-71e09da0-16ef-4930-9631-d4e00b9dbac7 None None] MFA Rules not processed for user `70ea2aaf529b46dabb70b71f4dcc3860`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:54:37.154075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-71e09da0-16ef-4930-9631-d4e00b9dbac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:54:37.163200 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 593/1197] 10.222.0.36 () {58 vars in 1058 bytes} [Mon Oct 31 14:54:37 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:54:37.170522 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ba5348-1dd6-4482-9dca-28fb617ac610 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:54:37.185092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ba5348-1dd6-4482-9dca-28fb617ac610 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:54:37.190253 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c66af3-af37-4ef7-b568-89114edeb415 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70ea2aaf529b46dabb70b71f4dcc3860', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:54:37.190883 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c66af3-af37-4ef7-b568-89114edeb415 service placement] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:37.191165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c66af3-af37-4ef7-b568-89114edeb415 service placement] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:37.191441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c66af3-af37-4ef7-b568-89114edeb415 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:37.200405 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c66af3-af37-4ef7-b568-89114edeb415 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:54:37.202009 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c66af3-af37-4ef7-b568-89114edeb415 service placement] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:54:37.214345 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 605/1198] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:54:37 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:54:44.758675 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66639615-3cd0-48fc-8843-ef77264a0c56 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:44.759232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66639615-3cd0-48fc-8843-ef77264a0c56 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:44.759414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66639615-3cd0-48fc-8843-ef77264a0c56 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:44.761080 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 594/1199] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 31 14:54:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:54:44.766091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8500b8fb-ff9e-4b3a-a188-277cbceedd8b None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:44.766500 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8500b8fb-ff9e-4b3a-a188-277cbceedd8b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:44.766677 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8500b8fb-ff9e-4b3a-a188-277cbceedd8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:44.787124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-8500b8fb-ff9e-4b3a-a188-277cbceedd8b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:54:44.788339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8500b8fb-ff9e-4b3a-a188-277cbceedd8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:54:44.797272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 606/1200] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 31 14:54:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:54:44.953460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66639615-3cd0-48fc-8843-ef77264a0c56 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:54:44.968581 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66639615-3cd0-48fc-8843-ef77264a0c56 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:54:44.973892 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea3f92a-272a-42b3-908a-1daf463a7a4d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:54:44.974478 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea3f92a-272a-42b3-908a-1daf463a7a4d service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:44.974755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea3f92a-272a-42b3-908a-1daf463a7a4d service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:44.975021 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea3f92a-272a-42b3-908a-1daf463a7a4d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:44.983143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea3f92a-272a-42b3-908a-1daf463a7a4d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:54:44.984758 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea3f92a-272a-42b3-908a-1daf463a7a4d service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:54:44.993953 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 595/1201] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:54:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:54:46.087727 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03ec76d-7d31-4191-849c-740ead2b2e6e None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:46.088301 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03ec76d-7d31-4191-849c-740ead2b2e6e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:46.088401 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03ec76d-7d31-4191-849c-740ead2b2e6e None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:46.089701 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 607/1202] 127.0.0.1 () {54 vars in 901 bytes} [Mon Oct 31 14:54:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:54:46.095173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb933911-f8de-4600-9f81-874517489a89 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:46.095600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb933911-f8de-4600-9f81-874517489a89 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:46.095746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb933911-f8de-4600-9f81-874517489a89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:46.115585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-bb933911-f8de-4600-9f81-874517489a89 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:54:46.116798 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-bb933911-f8de-4600-9f81-874517489a89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:54:46.124990 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 596/1203] 127.0.0.1 () {58 vars in 998 bytes} [Mon Oct 31 14:54:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:54:46.275662 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03ec76d-7d31-4191-849c-740ead2b2e6e None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:54:46.290738 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03ec76d-7d31-4191-849c-740ead2b2e6e None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:54:46.296923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c6d982-4b1e-4427-9f07-d411710fd330 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:54:46.297503 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c6d982-4b1e-4427-9f07-d411710fd330 service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:46.297784 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c6d982-4b1e-4427-9f07-d411710fd330 service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:46.298057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c6d982-4b1e-4427-9f07-d411710fd330 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:46.306079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c6d982-4b1e-4427-9f07-d411710fd330 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:54:46.307684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c6d982-4b1e-4427-9f07-d411710fd330 service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:54:46.317305 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 608/1204] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:54:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:54:50.377475 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ec7f14-9fa7-4372-ae46-e7885b104981 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:50.378040 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ec7f14-9fa7-4372-ae46-e7885b104981 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:50.378040 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ec7f14-9fa7-4372-ae46-e7885b104981 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:50.379350 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 597/1205] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 31 14:54:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:54:50.384442 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fc5d43-fa3b-4c7f-a84a-d9bac99ff7d2 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:50.384850 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fc5d43-fa3b-4c7f-a84a-d9bac99ff7d2 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:50.384983 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fc5d43-fa3b-4c7f-a84a-d9bac99ff7d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:50.405491 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-c1fc5d43-fa3b-4c7f-a84a-d9bac99ff7d2 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:54:50.406665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-c1fc5d43-fa3b-4c7f-a84a-d9bac99ff7d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:54:50.414353 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 609/1206] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 31 14:54:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:54:50.708824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ec7f14-9fa7-4372-ae46-e7885b104981 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:54:50.710547 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-85ec7f14-9fa7-4372-ae46-e7885b104981 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:54:50.794693 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ec7f14-9fa7-4372-ae46-e7885b104981 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:54:50.798358 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574afe32-013d-4d04-b645-dad9b39e573b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:54:50.798984 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574afe32-013d-4d04-b645-dad9b39e573b service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:50.799260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574afe32-013d-4d04-b645-dad9b39e573b service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:50.799529 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574afe32-013d-4d04-b645-dad9b39e573b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:50.808969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574afe32-013d-4d04-b645-dad9b39e573b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:54:50.810394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574afe32-013d-4d04-b645-dad9b39e573b service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:54:50.820599 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 598/1207] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:54:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:54:52.968144 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce483845-1129-430e-ae01-faab8ba6127b None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:52.968705 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce483845-1129-430e-ae01-faab8ba6127b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:52.968705 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce483845-1129-430e-ae01-faab8ba6127b None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:52.970192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 610/1208] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 31 14:54:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:54:52.975617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5c2cc4-945e-4677-8b01-a5906cc48a05 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:52.976078 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5c2cc4-945e-4677-8b01-a5906cc48a05 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:52.976181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5c2cc4-945e-4677-8b01-a5906cc48a05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:52.995832 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-7b5c2cc4-945e-4677-8b01-a5906cc48a05 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:54:52.997031 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-7b5c2cc4-945e-4677-8b01-a5906cc48a05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:54:53.004797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 599/1209] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 31 14:54:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:54:53.295420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce483845-1129-430e-ae01-faab8ba6127b None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:54:53.297166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ce483845-1129-430e-ae01-faab8ba6127b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:54:53.310003 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce483845-1129-430e-ae01-faab8ba6127b None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:54:53.313906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7502d177-d5bf-4790-9404-15e14475644c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:54:53.314501 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7502d177-d5bf-4790-9404-15e14475644c service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:53.314782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7502d177-d5bf-4790-9404-15e14475644c service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:53.315053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7502d177-d5bf-4790-9404-15e14475644c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:53.323767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7502d177-d5bf-4790-9404-15e14475644c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:54:53.325176 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7502d177-d5bf-4790-9404-15e14475644c service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:54:53.335512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 611/1210] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:54:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:54:57.349094 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09c87e9-6164-42b6-9655-5bea6041e136 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:57.349714 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09c87e9-6164-42b6-9655-5bea6041e136 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:57.349714 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09c87e9-6164-42b6-9655-5bea6041e136 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:57.351002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 600/1211] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 31 14:54:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:54:57.356286 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03742345-4465-4340-9134-eeb2c7ea0743 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:57.356737 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03742345-4465-4340-9134-eeb2c7ea0743 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:57.356834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03742345-4465-4340-9134-eeb2c7ea0743 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:57.365131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=81 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:54:57.378166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-03742345-4465-4340-9134-eeb2c7ea0743 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:54:57.379364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-03742345-4465-4340-9134-eeb2c7ea0743 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:54:57.387150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 612/1212] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 31 14:54:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:54:57.694738 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09c87e9-6164-42b6-9655-5bea6041e136 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:54:57.696439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e09c87e9-6164-42b6-9655-5bea6041e136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:54:57.761556 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:54:57.775342 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09c87e9-6164-42b6-9655-5bea6041e136 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:54:57.778886 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6348ec0-ed86-4761-b694-0f5c15c582ee service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:54:57.780382 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6348ec0-ed86-4761-b694-0f5c15c582ee service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:54:57.780658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6348ec0-ed86-4761-b694-0f5c15c582ee service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:54:57.780929 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6348ec0-ed86-4761-b694-0f5c15c582ee service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:54:57.789669 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6348ec0-ed86-4761-b694-0f5c15c582ee service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:54:57.791079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6348ec0-ed86-4761-b694-0f5c15c582ee service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:54:57.800314 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 601/1213] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:54:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:54:59.999422 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0d9200-b5ff-488c-917b-1327965f2ef2 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:55:00.000059 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0d9200-b5ff-488c-917b-1327965f2ef2 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:55:00.000059 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0d9200-b5ff-488c-917b-1327965f2ef2 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:55:00.001407 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 613/1214] 127.0.0.1 () {54 vars in 917 bytes} [Mon Oct 31 14:54:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:55:00.006727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c6d52e-7784-4b60-b00c-b35e94761925 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:55:00.007189 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c6d52e-7784-4b60-b00c-b35e94761925 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:55:00.007330 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c6d52e-7784-4b60-b00c-b35e94761925 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:55:00.027525 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-d5c6d52e-7784-4b60-b00c-b35e94761925 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:55:00.028732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d5c6d52e-7784-4b60-b00c-b35e94761925 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:55:00.036588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 602/1215] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Oct 31 14:55:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:55:00.337963 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0d9200-b5ff-488c-917b-1327965f2ef2 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:55:00.339672 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-aa0d9200-b5ff-488c-917b-1327965f2ef2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:55:00.352879 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0d9200-b5ff-488c-917b-1327965f2ef2 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:55:00.356376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dccdd93-a425-407f-b5a9-91ecd07fa8f9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:55:00.356956 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dccdd93-a425-407f-b5a9-91ecd07fa8f9 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:55:00.357429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dccdd93-a425-407f-b5a9-91ecd07fa8f9 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:55:00.357714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dccdd93-a425-407f-b5a9-91ecd07fa8f9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:55:00.366282 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dccdd93-a425-407f-b5a9-91ecd07fa8f9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:55:00.367718 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dccdd93-a425-407f-b5a9-91ecd07fa8f9 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:55:00.377283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 614/1216] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:55:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:55:10.022806 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=11 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:55:10.372841 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=8 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:24.962296 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec379b6-5019-4812-9ec0-d34c5a7c7f9a None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:24.962957 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec379b6-5019-4812-9ec0-d34c5a7c7f9a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:24.963063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec379b6-5019-4812-9ec0-d34c5a7c7f9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:24.973311 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca60d8-f1f2-421d-8596-d527b03b9deb None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:24.973926 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca60d8-f1f2-421d-8596-d527b03b9deb None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:24.974060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca60d8-f1f2-421d-8596-d527b03b9deb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:24.998927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-76ca60d8-f1f2-421d-8596-d527b03b9deb None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:25.000250 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-76ca60d8-f1f2-421d-8596-d527b03b9deb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:25.001317 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-5ec379b6-5019-4812-9ec0-d34c5a7c7f9a None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:25.002626 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5ec379b6-5019-4812-9ec0-d34c5a7c7f9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:25.013729 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 603/1217] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:25.017319 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 615/1218] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:56:25.035906 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231a4c0a-1b76-4f3b-98b6-a09fa75e2c05 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.036112 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7691e7-3704-4f89-8614-c8d20f0893a3 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.036242 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231a4c0a-1b76-4f3b-98b6-a09fa75e2c05 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.036485 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7691e7-3704-4f89-8614-c8d20f0893a3 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.036600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231a4c0a-1b76-4f3b-98b6-a09fa75e2c05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.036687 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7691e7-3704-4f89-8614-c8d20f0893a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.068990 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-231a4c0a-1b76-4f3b-98b6-a09fa75e2c05 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:25.070250 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-231a4c0a-1b76-4f3b-98b6-a09fa75e2c05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:25.072991 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-5c7691e7-3704-4f89-8614-c8d20f0893a3 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:25.074480 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5c7691e7-3704-4f89-8614-c8d20f0893a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:25.090409 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 604/1219] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:25.091589 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 616/1220] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:56:25.099212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7691e7-3704-4f89-8614-c8d20f0893a3 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.099454 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231a4c0a-1b76-4f3b-98b6-a09fa75e2c05 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.114268 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231a4c0a-1b76-4f3b-98b6-a09fa75e2c05 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.114616 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7691e7-3704-4f89-8614-c8d20f0893a3 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.121527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081b52cc-0e33-49be-b49e-3d70f75a8cdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.122337 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081b52cc-0e33-49be-b49e-3d70f75a8cdf admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.122775 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081b52cc-0e33-49be-b49e-3d70f75a8cdf admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.123057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081b52cc-0e33-49be-b49e-3d70f75a8cdf admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.123785 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081b52cc-0e33-49be-b49e-3d70f75a8cdf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:56:25.124094 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081b52cc-0e33-49be-b49e-3d70f75a8cdf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.125081 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c80105-d3c2-44cd-b024-8ef8d8fdbb14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.125464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: WARNING py.warnings [None req-081b52cc-0e33-49be-b49e-3d70f75a8cdf admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 31 14:56:25.125464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: warnings.warn(msg) Oct 31 14:56:25.125464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: Oct 31 14:56:25.125860 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c80105-d3c2-44cd-b024-8ef8d8fdbb14 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.126144 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c80105-d3c2-44cd-b024-8ef8d8fdbb14 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.126368 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081b52cc-0e33-49be-b49e-3d70f75a8cdf admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.126485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c80105-d3c2-44cd-b024-8ef8d8fdbb14 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.127047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c80105-d3c2-44cd-b024-8ef8d8fdbb14 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:56:25.127345 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c80105-d3c2-44cd-b024-8ef8d8fdbb14 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.129173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c80105-d3c2-44cd-b024-8ef8d8fdbb14 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.144297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 617/1221] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:56:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:56:25.148271 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 605/1222] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:56:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:56:25.165102 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081b52cc-0e33-49be-b49e-3d70f75a8cdf admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.165350 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c80105-d3c2-44cd-b024-8ef8d8fdbb14 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.178069 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081b52cc-0e33-49be-b49e-3d70f75a8cdf admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.181402 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c80105-d3c2-44cd-b024-8ef8d8fdbb14 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.189917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45173da-03cf-493c-b0d3-b24a82606093 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.190532 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45173da-03cf-493c-b0d3-b24a82606093 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.190816 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45173da-03cf-493c-b0d3-b24a82606093 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.191203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45173da-03cf-493c-b0d3-b24a82606093 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.191817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34cb2c9-e91b-4e9e-a81a-ecedf0bc62e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.191959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45173da-03cf-493c-b0d3-b24a82606093 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.192435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34cb2c9-e91b-4e9e-a81a-ecedf0bc62e6 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.192715 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34cb2c9-e91b-4e9e-a81a-ecedf0bc62e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.193120 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34cb2c9-e91b-4e9e-a81a-ecedf0bc62e6 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.193759 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34cb2c9-e91b-4e9e-a81a-ecedf0bc62e6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.193946 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45173da-03cf-493c-b0d3-b24a82606093 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.195017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: WARNING py.warnings [None req-a34cb2c9-e91b-4e9e-a81a-ecedf0bc62e6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 31 14:56:25.195017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: warnings.warn(msg) Oct 31 14:56:25.195017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: Oct 31 14:56:25.196494 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34cb2c9-e91b-4e9e-a81a-ecedf0bc62e6 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.254205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 606/1223] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/projects => generated 371 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:25.258675 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34cb2c9-e91b-4e9e-a81a-ecedf0bc62e6 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.262437 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 618/1224] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/projects => generated 377 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:25.267998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45173da-03cf-493c-b0d3-b24a82606093 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.364888 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34cb2c9-e91b-4e9e-a81a-ecedf0bc62e6 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.376723 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45173da-03cf-493c-b0d3-b24a82606093 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.388274 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434718a3-6a20-42ad-8865-b48afaf3493f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.388773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ade5ec6-ba66-4f43-b89c-b2f896518390 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.388914 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434718a3-6a20-42ad-8865-b48afaf3493f admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.389255 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434718a3-6a20-42ad-8865-b48afaf3493f admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.389470 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ade5ec6-ba66-4f43-b89c-b2f896518390 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.389649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434718a3-6a20-42ad-8865-b48afaf3493f admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.389747 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ade5ec6-ba66-4f43-b89c-b2f896518390 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.390156 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ade5ec6-ba66-4f43-b89c-b2f896518390 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.390283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434718a3-6a20-42ad-8865-b48afaf3493f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.390885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ade5ec6-ba66-4f43-b89c-b2f896518390 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.391937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: WARNING py.warnings [None req-7ade5ec6-ba66-4f43-b89c-b2f896518390 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 31 14:56:25.391937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: warnings.warn(msg) Oct 31 14:56:25.391937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: Oct 31 14:56:25.392552 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434718a3-6a20-42ad-8865-b48afaf3493f admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.393371 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ade5ec6-ba66-4f43-b89c-b2f896518390 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.430290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 619/1225] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/users => generated 289 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:25.433290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e87a26-f99d-4fb3-9de8-269568aaa6e2 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.433511 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 607/1226] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/users => generated 286 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:25.433912 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e87a26-f99d-4fb3-9de8-269568aaa6e2 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.434203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e87a26-f99d-4fb3-9de8-269568aaa6e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.434746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d12a54-e000-4b9a-9919-38a627c858c7 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.435031 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d12a54-e000-4b9a-9919-38a627c858c7 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.435301 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d12a54-e000-4b9a-9919-38a627c858c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.464861 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-b5e87a26-f99d-4fb3-9de8-269568aaa6e2 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:25.466513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b5e87a26-f99d-4fb3-9de8-269568aaa6e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:25.476559 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 620/1227] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:25.476838 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-70d12a54-e000-4b9a-9919-38a627c858c7 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:25.477779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2148f65a-e2e4-4fc6-aefe-09303c7c9980 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.477898 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-70d12a54-e000-4b9a-9919-38a627c858c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:25.478058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2148f65a-e2e4-4fc6-aefe-09303c7c9980 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.478333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2148f65a-e2e4-4fc6-aefe-09303c7c9980 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.486855 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 608/1228] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:25.487696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d12a54-e000-4b9a-9919-38a627c858c7 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.497278 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d12a54-e000-4b9a-9919-38a627c858c7 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.500912 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-2148f65a-e2e4-4fc6-aefe-09303c7c9980 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:25.502090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521de7d1-1518-4322-bab1-edb7280b7179 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.502230 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2148f65a-e2e4-4fc6-aefe-09303c7c9980 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:25.502690 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521de7d1-1518-4322-bab1-edb7280b7179 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.502968 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521de7d1-1518-4322-bab1-edb7280b7179 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.503239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521de7d1-1518-4322-bab1-edb7280b7179 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.503907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521de7d1-1518-4322-bab1-edb7280b7179 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.505492 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521de7d1-1518-4322-bab1-edb7280b7179 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.510363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 621/1229] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:25.511152 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2148f65a-e2e4-4fc6-aefe-09303c7c9980 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.521055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 609/1230] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:25 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:25.522293 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d41812-a26b-4059-b792-7638da25a02a None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.522594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d41812-a26b-4059-b792-7638da25a02a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.522859 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d41812-a26b-4059-b792-7638da25a02a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.527382 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2148f65a-e2e4-4fc6-aefe-09303c7c9980 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.533713 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3a6981-9342-41f7-96f6-4f1e354ffd87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.534331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3a6981-9342-41f7-96f6-4f1e354ffd87 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.534643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3a6981-9342-41f7-96f6-4f1e354ffd87 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.534889 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3a6981-9342-41f7-96f6-4f1e354ffd87 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.535541 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3a6981-9342-41f7-96f6-4f1e354ffd87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.537176 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3a6981-9342-41f7-96f6-4f1e354ffd87 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.543774 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-50d41812-a26b-4059-b792-7638da25a02a None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:25.545055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-50d41812-a26b-4059-b792-7638da25a02a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:25.551030 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 622/1231] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:25 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:25.551030 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdfa530-2075-40b2-8a18-ca10ce26c4e9 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.551030 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdfa530-2075-40b2-8a18-ca10ce26c4e9 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.551030 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdfa530-2075-40b2-8a18-ca10ce26c4e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.553870 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 610/1232] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:25.555170 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73166f0e-4fff-472a-bde0-1515449bdd60 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.555447 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73166f0e-4fff-472a-bde0-1515449bdd60 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.555802 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73166f0e-4fff-472a-bde0-1515449bdd60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.575301 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-8cdfa530-2075-40b2-8a18-ca10ce26c4e9 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:25.575995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-73166f0e-4fff-472a-bde0-1515449bdd60 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:25.576567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8cdfa530-2075-40b2-8a18-ca10ce26c4e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:25.577219 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-73166f0e-4fff-472a-bde0-1515449bdd60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:25.584390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 623/1233] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:25.593421 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 611/1234] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:25.593552 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cede85cc-86dd-463d-9cc1-9f7e6e54a1ba None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.593552 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cede85cc-86dd-463d-9cc1-9f7e6e54a1ba None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.593705 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6306bb62-cda5-4506-b064-a6f210c0fc71 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.593804 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cede85cc-86dd-463d-9cc1-9f7e6e54a1ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.593903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6306bb62-cda5-4506-b064-a6f210c0fc71 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.593903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6306bb62-cda5-4506-b064-a6f210c0fc71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.608623 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-6306bb62-cda5-4506-b064-a6f210c0fc71 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:25.609843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6306bb62-cda5-4506-b064-a6f210c0fc71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:25.611384 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-cede85cc-86dd-463d-9cc1-9f7e6e54a1ba None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:25.612568 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-cede85cc-86dd-463d-9cc1-9f7e6e54a1ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:25.617815 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 612/1235] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:25.618990 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247cba12-41ce-4f6d-99cb-7a68b46965f5 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.619278 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247cba12-41ce-4f6d-99cb-7a68b46965f5 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.619583 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247cba12-41ce-4f6d-99cb-7a68b46965f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.621004 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 624/1236] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:25.621764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cede85cc-86dd-463d-9cc1-9f7e6e54a1ba None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.631902 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cede85cc-86dd-463d-9cc1-9f7e6e54a1ba None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.636728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45573f6-5d33-4570-b14a-bc7b10e055b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.637755 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45573f6-5d33-4570-b14a-bc7b10e055b7 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.638037 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45573f6-5d33-4570-b14a-bc7b10e055b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.638471 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45573f6-5d33-4570-b14a-bc7b10e055b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.639668 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45573f6-5d33-4570-b14a-bc7b10e055b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.641673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45573f6-5d33-4570-b14a-bc7b10e055b7 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.642746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-247cba12-41ce-4f6d-99cb-7a68b46965f5 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:25.643960 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-247cba12-41ce-4f6d-99cb-7a68b46965f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:25.651756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 625/1237] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:25.656730 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 613/1238] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:25.656874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45573f6-5d33-4570-b14a-bc7b10e055b7 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.656999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dd3d16-1433-44b5-81ba-944d8190d12d None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.656999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dd3d16-1433-44b5-81ba-944d8190d12d None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.656999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dd3d16-1433-44b5-81ba-944d8190d12d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.667024 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45573f6-5d33-4570-b14a-bc7b10e055b7 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.671844 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1913e62c-d604-4ee9-8681-a25179be438e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.672431 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1913e62c-d604-4ee9-8681-a25179be438e admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.672707 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1913e62c-d604-4ee9-8681-a25179be438e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.672976 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1913e62c-d604-4ee9-8681-a25179be438e admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.673635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1913e62c-d604-4ee9-8681-a25179be438e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.674235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-e8dd3d16-1433-44b5-81ba-944d8190d12d None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:25.675202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1913e62c-d604-4ee9-8681-a25179be438e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.675434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e8dd3d16-1433-44b5-81ba-944d8190d12d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:25.683303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 614/1239] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:25.684442 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c479d1-6738-4193-8d3a-d9348834d225 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.684727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c479d1-6738-4193-8d3a-d9348834d225 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.685000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c479d1-6738-4193-8d3a-d9348834d225 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.687613 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 626/1240] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:25 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:25.687613 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323c3379-0db7-421c-a595-e665f363013e None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.687975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323c3379-0db7-421c-a595-e665f363013e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.688159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323c3379-0db7-421c-a595-e665f363013e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.706135 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-e7c479d1-6738-4193-8d3a-d9348834d225 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:25.707328 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e7c479d1-6738-4193-8d3a-d9348834d225 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:25.707802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-323c3379-0db7-421c-a595-e665f363013e None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:25.708956 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-323c3379-0db7-421c-a595-e665f363013e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:25.715434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 615/1241] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:25.717270 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 627/1242] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:25.717979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c479d1-6738-4193-8d3a-d9348834d225 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.721782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323c3379-0db7-421c-a595-e665f363013e None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.728550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c479d1-6738-4193-8d3a-d9348834d225 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.730941 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323c3379-0db7-421c-a595-e665f363013e None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.733557 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f620d73d-d821-48ef-8539-7f7fb61cca10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.734194 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f620d73d-d821-48ef-8539-7f7fb61cca10 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.734540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f620d73d-d821-48ef-8539-7f7fb61cca10 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.734887 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f620d73d-d821-48ef-8539-7f7fb61cca10 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.735580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b06ae0-5cdf-4dbf-9df4-d2525d0272a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.735717 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f620d73d-d821-48ef-8539-7f7fb61cca10 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:56:25.736061 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f620d73d-d821-48ef-8539-7f7fb61cca10 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.736175 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b06ae0-5cdf-4dbf-9df4-d2525d0272a4 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.736429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b06ae0-5cdf-4dbf-9df4-d2525d0272a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.736700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b06ae0-5cdf-4dbf-9df4-d2525d0272a4 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.737328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b06ae0-5cdf-4dbf-9df4-d2525d0272a4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:56:25.737632 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b06ae0-5cdf-4dbf-9df4-d2525d0272a4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.738053 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f620d73d-d821-48ef-8539-7f7fb61cca10 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.739224 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b06ae0-5cdf-4dbf-9df4-d2525d0272a4 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.751018 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 616/1243] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:56:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:56:25.758520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 628/1244] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:56:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:56:25.758639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f620d73d-d821-48ef-8539-7f7fb61cca10 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.758733 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b06ae0-5cdf-4dbf-9df4-d2525d0272a4 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.762389 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b06ae0-5cdf-4dbf-9df4-d2525d0272a4 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.763416 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f620d73d-d821-48ef-8539-7f7fb61cca10 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.767820 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1181a8-e4ae-497d-84a2-b76d636e35ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.768572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1181a8-e4ae-497d-84a2-b76d636e35ed admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.768687 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ad3794-3f10-4d77-9f8e-fe82bd9f33ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.768854 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1181a8-e4ae-497d-84a2-b76d636e35ed admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.769131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1181a8-e4ae-497d-84a2-b76d636e35ed admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.769295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ad3794-3f10-4d77-9f8e-fe82bd9f33ef admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.769597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ad3794-3f10-4d77-9f8e-fe82bd9f33ef admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.769847 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ad3794-3f10-4d77-9f8e-fe82bd9f33ef admin admin] PATH_INFO: `/v3/projects/fdab2d4f39c64f239ded71cc9773c8bb/users/75d1901be8064ff5bf79f0268d589e10/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.769942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1181a8-e4ae-497d-84a2-b76d636e35ed admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:56:25.770231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1181a8-e4ae-497d-84a2-b76d636e35ed admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.771850 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1181a8-e4ae-497d-84a2-b76d636e35ed admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.783134 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:25.786457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ad3794-3f10-4d77-9f8e-fe82bd9f33ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=fdab2d4f39c64f239ded71cc9773c8bb, user_id=75d1901be8064ff5bf79f0268d589e10, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.786946 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 629/1245] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:56:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:56:25.788115 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1181a8-e4ae-497d-84a2-b76d636e35ed admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.789862 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ad3794-3f10-4d77-9f8e-fe82bd9f33ef admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.800242 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:25.800855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1181a8-e4ae-497d-84a2-b76d636e35ed admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.807899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 617/1246] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:25 2022] PUT /identity/v3/projects/fdab2d4f39c64f239ded71cc9773c8bb/users/75d1901be8064ff5bf79f0268d589e10/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:25.812492 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ad3794-3f10-4d77-9f8e-fe82bd9f33ef admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.858855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fed5ea-064c-4eb6-b503-10244b575398 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.859622 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fed5ea-064c-4eb6-b503-10244b575398 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.859904 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fed5ea-064c-4eb6-b503-10244b575398 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.860175 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fed5ea-064c-4eb6-b503-10244b575398 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.860811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fed5ea-064c-4eb6-b503-10244b575398 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:56:25.861115 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fed5ea-064c-4eb6-b503-10244b575398 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.862695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fed5ea-064c-4eb6-b503-10244b575398 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.874650 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 630/1247] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:56:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:56:25.875056 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ad3794-3f10-4d77-9f8e-fe82bd9f33ef admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.875908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fed5ea-064c-4eb6-b503-10244b575398 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.880310 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e11962-b694-4caf-b5b5-3b9c6a1b7050 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.880992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e11962-b694-4caf-b5b5-3b9c6a1b7050 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.881270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e11962-b694-4caf-b5b5-3b9c6a1b7050 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.881537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e11962-b694-4caf-b5b5-3b9c6a1b7050 admin admin] PATH_INFO: `/v3/projects/713e15a5407049b2a8efd3ec38cd26cf/users/78dbaf070ba94a0894cecc104ab26e42/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.887325 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fed5ea-064c-4eb6-b503-10244b575398 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.892102 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c2e20d-6ae1-4ff3-86ad-e91adb3439d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.892760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c2e20d-6ae1-4ff3-86ad-e91adb3439d0 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.893044 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c2e20d-6ae1-4ff3-86ad-e91adb3439d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.893344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c2e20d-6ae1-4ff3-86ad-e91adb3439d0 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.893987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c2e20d-6ae1-4ff3-86ad-e91adb3439d0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:56:25.894295 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c2e20d-6ae1-4ff3-86ad-e91adb3439d0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.895907 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c2e20d-6ae1-4ff3-86ad-e91adb3439d0 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.898249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e11962-b694-4caf-b5b5-3b9c6a1b7050 admin admin] RBAC: Authorizing `identity:create_grant(project_id=713e15a5407049b2a8efd3ec38cd26cf, user_id=78dbaf070ba94a0894cecc104ab26e42, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.900938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e11962-b694-4caf-b5b5-3b9c6a1b7050 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.909387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 631/1248] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:56:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:56:25.910215 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c2e20d-6ae1-4ff3-86ad-e91adb3439d0 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.915019 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 618/1249] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:25 2022] PUT /identity/v3/projects/713e15a5407049b2a8efd3ec38cd26cf/users/78dbaf070ba94a0894cecc104ab26e42/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:25.916633 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e11962-b694-4caf-b5b5-3b9c6a1b7050 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:25.925265 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c2e20d-6ae1-4ff3-86ad-e91adb3439d0 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.981743 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829d37f8-a8fe-4583-9e62-8ccab9f1ae1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.982452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829d37f8-a8fe-4583-9e62-8ccab9f1ae1d admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.982560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829d37f8-a8fe-4583-9e62-8ccab9f1ae1d admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.983332 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829d37f8-a8fe-4583-9e62-8ccab9f1ae1d admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.984078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829d37f8-a8fe-4583-9e62-8ccab9f1ae1d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:56:25.984212 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e11962-b694-4caf-b5b5-3b9c6a1b7050 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:25.984649 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829d37f8-a8fe-4583-9e62-8ccab9f1ae1d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.985981 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829d37f8-a8fe-4583-9e62-8ccab9f1ae1d admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:25.989137 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a617ec2a-57bd-4f0e-8d60-849a008e30c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:25.989769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a617ec2a-57bd-4f0e-8d60-849a008e30c2 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:25.990051 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a617ec2a-57bd-4f0e-8d60-849a008e30c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:25.990352 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a617ec2a-57bd-4f0e-8d60-849a008e30c2 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:25.991044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a617ec2a-57bd-4f0e-8d60-849a008e30c2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:25.992973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a617ec2a-57bd-4f0e-8d60-849a008e30c2 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.006598 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 632/1250] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:56:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:56:26.006598 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829d37f8-a8fe-4583-9e62-8ccab9f1ae1d admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.010248 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829d37f8-a8fe-4583-9e62-8ccab9f1ae1d admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.016099 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e6cdc7-c6fd-4e4d-883e-f17c0bcd0a6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.016679 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e6cdc7-c6fd-4e4d-883e-f17c0bcd0a6a admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.016960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e6cdc7-c6fd-4e4d-883e-f17c0bcd0a6a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.017235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e6cdc7-c6fd-4e4d-883e-f17c0bcd0a6a admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.017880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e6cdc7-c6fd-4e4d-883e-f17c0bcd0a6a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.020005 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e6cdc7-c6fd-4e4d-883e-f17c0bcd0a6a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.024779 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 619/1251] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/projects => generated 385 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:26.025834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a617ec2a-57bd-4f0e-8d60-849a008e30c2 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.055809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 633/1252] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:56:25 2022] POST /identity/v3/projects => generated 389 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:26.055809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baa7596-2d4d-4d6b-923b-ef9dc17dbf3a None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.055809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baa7596-2d4d-4d6b-923b-ef9dc17dbf3a None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.055809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baa7596-2d4d-4d6b-923b-ef9dc17dbf3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.073441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-0baa7596-2d4d-4d6b-923b-ef9dc17dbf3a None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:26.074634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0baa7596-2d4d-4d6b-923b-ef9dc17dbf3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:26.091200 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a617ec2a-57bd-4f0e-8d60-849a008e30c2 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.100660 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:26.123797 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:26.138214 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 634/1253] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:26.141520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0baa7596-2d4d-4d6b-923b-ef9dc17dbf3a None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.148772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0baa7596-2d4d-4d6b-923b-ef9dc17dbf3a None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.153311 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ad6b9e-e9af-4542-98d1-cfbed3c23a5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.153906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ad6b9e-e9af-4542-98d1-cfbed3c23a5d admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.154184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ad6b9e-e9af-4542-98d1-cfbed3c23a5d admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.154456 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ad6b9e-e9af-4542-98d1-cfbed3c23a5d admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.155322 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd32197e-7f16-4cae-8294-594cec3a67dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.155704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ad6b9e-e9af-4542-98d1-cfbed3c23a5d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.155968 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd32197e-7f16-4cae-8294-594cec3a67dc admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.156247 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd32197e-7f16-4cae-8294-594cec3a67dc admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.156519 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd32197e-7f16-4cae-8294-594cec3a67dc admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.157145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd32197e-7f16-4cae-8294-594cec3a67dc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.157771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ad6b9e-e9af-4542-98d1-cfbed3c23a5d admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.159034 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd32197e-7f16-4cae-8294-594cec3a67dc admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.190344 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 620/1254] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:56:26 2022] POST /identity/v3/projects => generated 405 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:26.190572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 635/1255] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:56:26 2022] POST /identity/v3/projects => generated 393 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:26.190683 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd32197e-7f16-4cae-8294-594cec3a67dc admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.190802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937385e5-a745-4d7e-9d12-ba5aaa8cd531 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.190802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937385e5-a745-4d7e-9d12-ba5aaa8cd531 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.190802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937385e5-a745-4d7e-9d12-ba5aaa8cd531 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.219728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-937385e5-a745-4d7e-9d12-ba5aaa8cd531 None None] MFA Rules not processed for user `78dbaf070ba94a0894cecc104ab26e42`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:26.220962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-937385e5-a745-4d7e-9d12-ba5aaa8cd531 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:26.257097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd32197e-7f16-4cae-8294-594cec3a67dc admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.287737 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 636/1256] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:26 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:26.288567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937385e5-a745-4d7e-9d12-ba5aaa8cd531 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.315926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b026f90-d76b-4273-a2d3-630a8a8dd88c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.316907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b026f90-d76b-4273-a2d3-630a8a8dd88c admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.317285 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b026f90-d76b-4273-a2d3-630a8a8dd88c admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.317710 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b026f90-d76b-4273-a2d3-630a8a8dd88c admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.318446 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b026f90-d76b-4273-a2d3-630a8a8dd88c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.321928 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b026f90-d76b-4273-a2d3-630a8a8dd88c admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.356310 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 621/1257] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:56:26 2022] POST /identity/v3/projects => generated 379 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:26.356310 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b026f90-d76b-4273-a2d3-630a8a8dd88c admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.359211 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937385e5-a745-4d7e-9d12-ba5aaa8cd531 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.387095 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:26.416095 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726628e8-5270-4c79-8adc-739b991e8b46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.416869 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726628e8-5270-4c79-8adc-739b991e8b46 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.417151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726628e8-5270-4c79-8adc-739b991e8b46 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.417408 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726628e8-5270-4c79-8adc-739b991e8b46 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.418037 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726628e8-5270-4c79-8adc-739b991e8b46 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.418260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b026f90-d76b-4273-a2d3-630a8a8dd88c admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.420165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726628e8-5270-4c79-8adc-739b991e8b46 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.423007 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c2c1df-478d-4a67-9ffa-fe84dd284e11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.423633 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c2c1df-478d-4a67-9ffa-fe84dd284e11 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.423913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c2c1df-478d-4a67-9ffa-fe84dd284e11 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.424182 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c2c1df-478d-4a67-9ffa-fe84dd284e11 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.425012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c2c1df-478d-4a67-9ffa-fe84dd284e11 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.427245 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c2c1df-478d-4a67-9ffa-fe84dd284e11 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.434168 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:26.450816 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 637/1258] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:56:26 2022] POST /identity/v3/projects => generated 379 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:26.450816 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726628e8-5270-4c79-8adc-739b991e8b46 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.451126 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 622/1259] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:56:26 2022] POST /identity/v3/users => generated 293 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:26.451126 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c2c1df-478d-4a67-9ffa-fe84dd284e11 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.507602 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c2c1df-478d-4a67-9ffa-fe84dd284e11 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.513488 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726628e8-5270-4c79-8adc-739b991e8b46 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.519054 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5600d8af-97ca-4e9a-b82c-6a4399e3af04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.519691 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5600d8af-97ca-4e9a-b82c-6a4399e3af04 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.519972 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5600d8af-97ca-4e9a-b82c-6a4399e3af04 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.520277 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5600d8af-97ca-4e9a-b82c-6a4399e3af04 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.520910 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5600d8af-97ca-4e9a-b82c-6a4399e3af04 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.522817 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5600d8af-97ca-4e9a-b82c-6a4399e3af04 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.546248 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 638/1260] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:56:26 2022] POST /identity/v3/users => generated 295 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:26.547176 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5600d8af-97ca-4e9a-b82c-6a4399e3af04 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.556700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5600d8af-97ca-4e9a-b82c-6a4399e3af04 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.561515 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229649df-12e6-48c4-8640-dd2b8350c12d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.562146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229649df-12e6-48c4-8640-dd2b8350c12d admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.562443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229649df-12e6-48c4-8640-dd2b8350c12d admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.562717 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229649df-12e6-48c4-8640-dd2b8350c12d admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.563340 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229649df-12e6-48c4-8640-dd2b8350c12d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.565402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229649df-12e6-48c4-8640-dd2b8350c12d admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.566649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fca922-326e-4dfb-bb0c-98436d4330bd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.567255 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fca922-326e-4dfb-bb0c-98436d4330bd service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.567529 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fca922-326e-4dfb-bb0c-98436d4330bd service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.567835 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fca922-326e-4dfb-bb0c-98436d4330bd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.577836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fca922-326e-4dfb-bb0c-98436d4330bd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.579633 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fca922-326e-4dfb-bb0c-98436d4330bd service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.587474 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 639/1261] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:56:26 2022] POST /identity/v3/users => generated 303 bytes in 41 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:26.588306 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229649df-12e6-48c4-8640-dd2b8350c12d admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.590938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 623/1262] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:26.592234 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f6e601-2b91-46f5-ade3-7d46e6e96ae7 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.593023 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f6e601-2b91-46f5-ade3-7d46e6e96ae7 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.593296 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f6e601-2b91-46f5-ade3-7d46e6e96ae7 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.594539 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 624/1263] 10.222.0.36 () {54 vars in 961 bytes} [Mon Oct 31 14:56:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:56:26.595252 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f6e601-2b91-46f5-ade3-7d46e6e96ae7 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.598781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229649df-12e6-48c4-8640-dd2b8350c12d admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.605106 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee3be60-7897-4a58-a31c-76d73256c982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.605631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee3be60-7897-4a58-a31c-76d73256c982 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.605974 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee3be60-7897-4a58-a31c-76d73256c982 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.606314 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee3be60-7897-4a58-a31c-76d73256c982 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.607009 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee3be60-7897-4a58-a31c-76d73256c982 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.608075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f6e601-2b91-46f5-ade3-7d46e6e96ae7 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.609398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee3be60-7897-4a58-a31c-76d73256c982 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.613407 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92aea5b-01aa-4efd-8928-8df68cf4df26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.614075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92aea5b-01aa-4efd-8928-8df68cf4df26 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.614421 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92aea5b-01aa-4efd-8928-8df68cf4df26 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.614758 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92aea5b-01aa-4efd-8928-8df68cf4df26 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.615449 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92aea5b-01aa-4efd-8928-8df68cf4df26 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.617499 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92aea5b-01aa-4efd-8928-8df68cf4df26 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.635940 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 640/1264] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:56:26 2022] POST /identity/v3/users => generated 297 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:26.637169 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee3be60-7897-4a58-a31c-76d73256c982 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.642617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 625/1265] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:56:26 2022] POST /identity/v3/users => generated 290 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:26.643398 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92aea5b-01aa-4efd-8928-8df68cf4df26 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.651615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee3be60-7897-4a58-a31c-76d73256c982 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.653277 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92aea5b-01aa-4efd-8928-8df68cf4df26 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.657282 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7593f68-cf50-43fb-b138-08c4ccb018b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.657896 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7593f68-cf50-43fb-b138-08c4ccb018b3 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.658177 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7593f68-cf50-43fb-b138-08c4ccb018b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.658452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7593f68-cf50-43fb-b138-08c4ccb018b3 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.659085 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7593f68-cf50-43fb-b138-08c4ccb018b3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.659457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e122cc-5cef-4e8c-991d-112940a247f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.660601 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e122cc-5cef-4e8c-991d-112940a247f5 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.660981 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e122cc-5cef-4e8c-991d-112940a247f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.661108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7593f68-cf50-43fb-b138-08c4ccb018b3 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.661362 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e122cc-5cef-4e8c-991d-112940a247f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.662134 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e122cc-5cef-4e8c-991d-112940a247f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.664109 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e122cc-5cef-4e8c-991d-112940a247f5 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.676375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 626/1266] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:26 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:26.677103 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e122cc-5cef-4e8c-991d-112940a247f5 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.688019 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e122cc-5cef-4e8c-991d-112940a247f5 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.688647 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 641/1267] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:56:26 2022] POST /identity/v3/users => generated 290 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:26.689726 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7593f68-cf50-43fb-b138-08c4ccb018b3 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.696670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b18e02-e8fa-4746-9924-a0a66e820b1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.697291 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b18e02-e8fa-4746-9924-a0a66e820b1b admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.697572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b18e02-e8fa-4746-9924-a0a66e820b1b admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.697844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b18e02-e8fa-4746-9924-a0a66e820b1b admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.698491 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b18e02-e8fa-4746-9924-a0a66e820b1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.700129 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b18e02-e8fa-4746-9924-a0a66e820b1b admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.701031 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7593f68-cf50-43fb-b138-08c4ccb018b3 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.706443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e659c2-8ddd-4117-a72a-b07d7b5111d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.707450 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e659c2-8ddd-4117-a72a-b07d7b5111d6 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.707450 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e659c2-8ddd-4117-a72a-b07d7b5111d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.707737 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e659c2-8ddd-4117-a72a-b07d7b5111d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.708255 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e659c2-8ddd-4117-a72a-b07d7b5111d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.709846 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e659c2-8ddd-4117-a72a-b07d7b5111d6 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.716144 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 627/1268] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:26 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:26.716144 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba4c2eb-2457-47a8-8678-241335ca644b None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.716144 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba4c2eb-2457-47a8-8678-241335ca644b None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.716144 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba4c2eb-2457-47a8-8678-241335ca644b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.724838 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 642/1269] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:26 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:26.725837 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e659c2-8ddd-4117-a72a-b07d7b5111d6 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.735925 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e659c2-8ddd-4117-a72a-b07d7b5111d6 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.740865 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f14f237-4465-4153-9b6d-a73c493960de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.741480 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f14f237-4465-4153-9b6d-a73c493960de admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.741765 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f14f237-4465-4153-9b6d-a73c493960de admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.742023 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f14f237-4465-4153-9b6d-a73c493960de admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.742662 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f14f237-4465-4153-9b6d-a73c493960de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.744309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f14f237-4465-4153-9b6d-a73c493960de admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.755887 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 643/1270] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:26 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:26.757094 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f14f237-4465-4153-9b6d-a73c493960de admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.764839 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-fba4c2eb-2457-47a8-8678-241335ca644b None None] MFA Rules not processed for user `c8cace029a92406f850bb0a4ca07c461`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:26.766319 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-fba4c2eb-2457-47a8-8678-241335ca644b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:26.769859 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f14f237-4465-4153-9b6d-a73c493960de admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.777029 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a582ea-06b8-4e67-b7ce-aea669593460 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.778161 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a582ea-06b8-4e67-b7ce-aea669593460 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.778735 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a582ea-06b8-4e67-b7ce-aea669593460 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.779353 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a582ea-06b8-4e67-b7ce-aea669593460 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.781338 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a582ea-06b8-4e67-b7ce-aea669593460 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.785550 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a582ea-06b8-4e67-b7ce-aea669593460 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.800443 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:26.804545 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 644/1271] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:26 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:26.808871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a582ea-06b8-4e67-b7ce-aea669593460 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.822716 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a582ea-06b8-4e67-b7ce-aea669593460 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.828639 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194fa641-7bf5-477c-973b-4ba9d3e7227f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.829256 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194fa641-7bf5-477c-973b-4ba9d3e7227f admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.829541 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194fa641-7bf5-477c-973b-4ba9d3e7227f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.829827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194fa641-7bf5-477c-973b-4ba9d3e7227f admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.830462 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194fa641-7bf5-477c-973b-4ba9d3e7227f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.832257 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194fa641-7bf5-477c-973b-4ba9d3e7227f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.844514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 645/1272] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:26 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:26.845375 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194fa641-7bf5-477c-973b-4ba9d3e7227f admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.855795 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194fa641-7bf5-477c-973b-4ba9d3e7227f admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.861683 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8798e5-d85c-46b3-b211-0cac14fc99fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.862588 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8798e5-d85c-46b3-b211-0cac14fc99fb admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.863023 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8798e5-d85c-46b3-b211-0cac14fc99fb admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.863447 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8798e5-d85c-46b3-b211-0cac14fc99fb admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.864257 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca8798e5-d85c-46b3-b211-0cac14fc99fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.864505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 628/1273] 10.222.0.36 () {58 vars in 1058 bytes} [Mon Oct 31 14:56:26 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:26.865378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba4c2eb-2457-47a8-8678-241335ca644b None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.866538 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca8798e5-d85c-46b3-b211-0cac14fc99fb admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.876723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba4c2eb-2457-47a8-8678-241335ca644b None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.884566 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 646/1274] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:26 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:26.884566 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8798e5-d85c-46b3-b211-0cac14fc99fb admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.886855 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f01006-5ced-4cc2-a962-502fcec01beb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.887540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f01006-5ced-4cc2-a962-502fcec01beb admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.887795 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f01006-5ced-4cc2-a962-502fcec01beb admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.888065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f01006-5ced-4cc2-a962-502fcec01beb admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.889180 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f01006-5ced-4cc2-a962-502fcec01beb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.891051 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f01006-5ced-4cc2-a962-502fcec01beb admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.900730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8798e5-d85c-46b3-b211-0cac14fc99fb admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.907108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 629/1275] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:26 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:26.907108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f01006-5ced-4cc2-a962-502fcec01beb admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.912658 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8142ca2-45ba-454f-8e2b-56cbfa5877cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.914032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8142ca2-45ba-454f-8e2b-56cbfa5877cb admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.915259 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8142ca2-45ba-454f-8e2b-56cbfa5877cb admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.915880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8142ca2-45ba-454f-8e2b-56cbfa5877cb admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.917005 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8142ca2-45ba-454f-8e2b-56cbfa5877cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.918810 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8142ca2-45ba-454f-8e2b-56cbfa5877cb admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.919143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f01006-5ced-4cc2-a962-502fcec01beb admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.925341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e541287-acb5-4fe4-9539-4210d4364ef9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.925954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e541287-acb5-4fe4-9539-4210d4364ef9 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.926293 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e541287-acb5-4fe4-9539-4210d4364ef9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.926574 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e541287-acb5-4fe4-9539-4210d4364ef9 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.927274 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e541287-acb5-4fe4-9539-4210d4364ef9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.928970 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e541287-acb5-4fe4-9539-4210d4364ef9 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.930776 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 647/1276] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:26 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:26.931636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8142ca2-45ba-454f-8e2b-56cbfa5877cb admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.946364 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 630/1277] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:26 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:26.946364 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e541287-acb5-4fe4-9539-4210d4364ef9 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.946660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8142ca2-45ba-454f-8e2b-56cbfa5877cb admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.949909 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e934665-fe2f-43b6-a37f-6264bd117187 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.950514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e934665-fe2f-43b6-a37f-6264bd117187 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.950918 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e934665-fe2f-43b6-a37f-6264bd117187 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.951377 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e934665-fe2f-43b6-a37f-6264bd117187 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.952640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e934665-fe2f-43b6-a37f-6264bd117187 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:26.954075 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e934665-fe2f-43b6-a37f-6264bd117187 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:26.958193 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e541287-acb5-4fe4-9539-4210d4364ef9 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:26.969495 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf70ea63-dbc1-49e5-afbd-3e7df9bf2c94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:26.970317 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf70ea63-dbc1-49e5-afbd-3e7df9bf2c94 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:26.970671 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf70ea63-dbc1-49e5-afbd-3e7df9bf2c94 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:26.971081 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf70ea63-dbc1-49e5-afbd-3e7df9bf2c94 admin admin] PATH_INFO: `/v3/projects/f629aa044ca845069237ecb8d889284e/users/cdcbcb2ed9e44dbe8444e531fc1eb707/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:26.982591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 648/1278] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:26 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:26.984748 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e934665-fe2f-43b6-a37f-6264bd117187 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:26.999739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e934665-fe2f-43b6-a37f-6264bd117187 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:27.005247 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf70ea63-dbc1-49e5-afbd-3e7df9bf2c94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f629aa044ca845069237ecb8d889284e, user_id=cdcbcb2ed9e44dbe8444e531fc1eb707, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:27.008631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08245cf5-dfd5-40fc-b142-b3143da435cd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:27.008864 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf70ea63-dbc1-49e5-afbd-3e7df9bf2c94 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:27.009786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08245cf5-dfd5-40fc-b142-b3143da435cd service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.010275 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08245cf5-dfd5-40fc-b142-b3143da435cd service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.010935 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08245cf5-dfd5-40fc-b142-b3143da435cd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.026099 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:INSERT=13,keystone:SELECT=87 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:27.027636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 631/1279] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:26 2022] PUT /identity/v3/projects/f629aa044ca845069237ecb8d889284e/users/cdcbcb2ed9e44dbe8444e531fc1eb707/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:27.029044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf70ea63-dbc1-49e5-afbd-3e7df9bf2c94 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:27.095660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08245cf5-dfd5-40fc-b142-b3143da435cd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:27.097653 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08245cf5-dfd5-40fc-b142-b3143da435cd service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:27.109493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 649/1280] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:26 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:27.110329 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08245cf5-dfd5-40fc-b142-b3143da435cd service nova] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:27.113377 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf70ea63-dbc1-49e5-afbd-3e7df9bf2c94 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:27.121091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244647e3-f22b-487c-9cba-a49f7102432c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:27.121789 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244647e3-f22b-487c-9cba-a49f7102432c admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.122137 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244647e3-f22b-487c-9cba-a49f7102432c admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.122279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08245cf5-dfd5-40fc-b142-b3143da435cd service nova] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:27.122537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244647e3-f22b-487c-9cba-a49f7102432c admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.123262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244647e3-f22b-487c-9cba-a49f7102432c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:27.124983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244647e3-f22b-487c-9cba-a49f7102432c admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:27.128343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761adeb4-d4a4-409f-b4c1-56541d99d3aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:27.129079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761adeb4-d4a4-409f-b4c1-56541d99d3aa admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.129606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761adeb4-d4a4-409f-b4c1-56541d99d3aa admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.129957 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761adeb4-d4a4-409f-b4c1-56541d99d3aa admin admin] PATH_INFO: `/v3/projects/5119fe6e3c064da0a66218481a944aaa/users/e1fb0b1657cd43baaa93a16ce6d9ec8f/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.141273 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 632/1281] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:27 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:27.142253 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244647e3-f22b-487c-9cba-a49f7102432c admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:27.151657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761adeb4-d4a4-409f-b4c1-56541d99d3aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=5119fe6e3c064da0a66218481a944aaa, user_id=e1fb0b1657cd43baaa93a16ce6d9ec8f, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:27.154379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244647e3-f22b-487c-9cba-a49f7102432c admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:27.156434 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761adeb4-d4a4-409f-b4c1-56541d99d3aa admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:27.159852 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dcc0ca-21ad-40d7-aeba-09c99c893e0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:27.160720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dcc0ca-21ad-40d7-aeba-09c99c893e0a admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.161393 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dcc0ca-21ad-40d7-aeba-09c99c893e0a admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.161640 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dcc0ca-21ad-40d7-aeba-09c99c893e0a admin admin] PATH_INFO: `/v3/projects/d8dc1ffa3cab4805a4b7010ee4a5bc19/users/a38d8c2788cf40249c1daedc0ad4cdf6/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.175575 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 650/1282] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:27 2022] PUT /identity/v3/projects/5119fe6e3c064da0a66218481a944aaa/users/e1fb0b1657cd43baaa93a16ce6d9ec8f/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:27.176627 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761adeb4-d4a4-409f-b4c1-56541d99d3aa admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:27.188656 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dcc0ca-21ad-40d7-aeba-09c99c893e0a admin admin] RBAC: Authorizing `identity:create_grant(project_id=d8dc1ffa3cab4805a4b7010ee4a5bc19, user_id=a38d8c2788cf40249c1daedc0ad4cdf6, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:27.192816 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dcc0ca-21ad-40d7-aeba-09c99c893e0a admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:27.219225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 633/1283] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:27 2022] PUT /identity/v3/projects/d8dc1ffa3cab4805a4b7010ee4a5bc19/users/a38d8c2788cf40249c1daedc0ad4cdf6/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:27.220693 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dcc0ca-21ad-40d7-aeba-09c99c893e0a admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:27.261504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761adeb4-d4a4-409f-b4c1-56541d99d3aa admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:27.296649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:27.320487 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dcc0ca-21ad-40d7-aeba-09c99c893e0a admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:27.326113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aaa9cde-3fcf-4d68-92e0-e43539503184 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:27.328160 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aaa9cde-3fcf-4d68-92e0-e43539503184 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.328468 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aaa9cde-3fcf-4d68-92e0-e43539503184 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.335325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aaa9cde-3fcf-4d68-92e0-e43539503184 admin admin] PATH_INFO: `/v3/projects/009f7c73f7dc486880989c38883379dd/users/94d17cbf9c7d49da8cd289c0d1c390e8/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.360503 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2f518f-db80-4fd4-8789-949b2fb68cae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:27.361279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2f518f-db80-4fd4-8789-949b2fb68cae admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.361652 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2f518f-db80-4fd4-8789-949b2fb68cae admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.362020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2f518f-db80-4fd4-8789-949b2fb68cae admin admin] PATH_INFO: `/v3/projects/8bc7c0eaebf146bea1af6b4d708bf62b/users/e6bfe2c1ba054935ae8e9fcc054faf68/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.368475 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aaa9cde-3fcf-4d68-92e0-e43539503184 admin admin] RBAC: Authorizing `identity:create_grant(project_id=009f7c73f7dc486880989c38883379dd, user_id=94d17cbf9c7d49da8cd289c0d1c390e8, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:27.371402 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aaa9cde-3fcf-4d68-92e0-e43539503184 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:27.381761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2f518f-db80-4fd4-8789-949b2fb68cae admin admin] RBAC: Authorizing `identity:create_grant(project_id=8bc7c0eaebf146bea1af6b4d708bf62b, user_id=e6bfe2c1ba054935ae8e9fcc054faf68, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:27.385581 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2f518f-db80-4fd4-8789-949b2fb68cae admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:27.399052 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 634/1284] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:27 2022] PUT /identity/v3/projects/009f7c73f7dc486880989c38883379dd/users/94d17cbf9c7d49da8cd289c0d1c390e8/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:27.401957 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5238377d-f341-49cb-933d-a42c40ff8134 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.402243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5238377d-f341-49cb-933d-a42c40ff8134 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.402515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5238377d-f341-49cb-933d-a42c40ff8134 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.406934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 651/1285] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:27 2022] PUT /identity/v3/projects/8bc7c0eaebf146bea1af6b4d708bf62b/users/e6bfe2c1ba054935ae8e9fcc054faf68/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:27.408915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2f518f-db80-4fd4-8789-949b2fb68cae admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:27.442090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-5238377d-f341-49cb-933d-a42c40ff8134 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:27.443600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5238377d-f341-49cb-933d-a42c40ff8134 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:27.445890 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:27.521785 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2f518f-db80-4fd4-8789-949b2fb68cae admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:27.525657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2c0fdd-1e02-48cd-be14-67eb2b325e18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:27.526299 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 635/1286] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:27.526769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2c0fdd-1e02-48cd-be14-67eb2b325e18 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.527147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2c0fdd-1e02-48cd-be14-67eb2b325e18 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.527657 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa59db1-6705-42bc-a5a9-f53675058f22 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.527777 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2c0fdd-1e02-48cd-be14-67eb2b325e18 admin admin] PATH_INFO: `/v3/projects/af86d70d54694b7995d03c8c607ae706/users/77af489d5daa4d028b737c5f4efc93db/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.527946 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa59db1-6705-42bc-a5a9-f53675058f22 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.528224 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa59db1-6705-42bc-a5a9-f53675058f22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.552653 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-caa59db1-6705-42bc-a5a9-f53675058f22 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:27.554838 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2c0fdd-1e02-48cd-be14-67eb2b325e18 admin admin] RBAC: Authorizing `identity:create_grant(project_id=af86d70d54694b7995d03c8c607ae706, user_id=77af489d5daa4d028b737c5f4efc93db, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:27.555063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-caa59db1-6705-42bc-a5a9-f53675058f22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:27.558140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2c0fdd-1e02-48cd-be14-67eb2b325e18 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:27.566103 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 636/1287] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:27.567646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f588f1d0-1591-4b81-86b2-f31faf891a89 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.568032 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f588f1d0-1591-4b81-86b2-f31faf891a89 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.568432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f588f1d0-1591-4b81-86b2-f31faf891a89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.581789 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 652/1288] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:27 2022] PUT /identity/v3/projects/af86d70d54694b7995d03c8c607ae706/users/77af489d5daa4d028b737c5f4efc93db/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:27.585872 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d634b1-e2fa-41a0-8499-dbf3e1b30c27 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.585872 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d634b1-e2fa-41a0-8499-dbf3e1b30c27 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.585872 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d634b1-e2fa-41a0-8499-dbf3e1b30c27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.601925 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-f588f1d0-1591-4b81-86b2-f31faf891a89 None None] MFA Rules not processed for user `a38d8c2788cf40249c1daedc0ad4cdf6`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:27.603272 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-f588f1d0-1591-4b81-86b2-f31faf891a89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:27.618927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-66d634b1-e2fa-41a0-8499-dbf3e1b30c27 None None] MFA Rules not processed for user `94d17cbf9c7d49da8cd289c0d1c390e8`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:27.621046 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-66d634b1-e2fa-41a0-8499-dbf3e1b30c27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:27.693181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 637/1289] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:27 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:27.694508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f7c6bb-31c3-4234-a0cb-958e99c74be9 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.695017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f7c6bb-31c3-4234-a0cb-958e99c74be9 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.695403 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f7c6bb-31c3-4234-a0cb-958e99c74be9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.715477 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 653/1290] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:27 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:27.715477 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500e5da6-67d0-4dca-9db9-0a7c9c3406f0 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.715477 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500e5da6-67d0-4dca-9db9-0a7c9c3406f0 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.715477 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500e5da6-67d0-4dca-9db9-0a7c9c3406f0 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.716612 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 654/1291] 10.222.0.36 () {54 vars in 912 bytes} [Mon Oct 31 14:56:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:56:27.717714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500e5da6-67d0-4dca-9db9-0a7c9c3406f0 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:27.726988 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-49f7c6bb-31c3-4234-a0cb-958e99c74be9 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:27.729527 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-49f7c6bb-31c3-4234-a0cb-958e99c74be9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:27.787713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:27.799321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500e5da6-67d0-4dca-9db9-0a7c9c3406f0 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:27.805474 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4f801d-97b9-4293-855f-8a239b9cf718 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:27.806092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4f801d-97b9-4293-855f-8a239b9cf718 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.806393 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4f801d-97b9-4293-855f-8a239b9cf718 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.806772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4f801d-97b9-4293-855f-8a239b9cf718 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.811817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 638/1292] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:27.812988 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f7c6bb-31c3-4234-a0cb-958e99c74be9 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:27.817415 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4f801d-97b9-4293-855f-8a239b9cf718 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:27.818871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4f801d-97b9-4293-855f-8a239b9cf718 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:27.831193 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f7c6bb-31c3-4234-a0cb-958e99c74be9 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:27.835472 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51144c02-4182-4a2a-8b0c-edba111d5eb8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:27.836855 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51144c02-4182-4a2a-8b0c-edba111d5eb8 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.836855 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51144c02-4182-4a2a-8b0c-edba111d5eb8 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.836855 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51144c02-4182-4a2a-8b0c-edba111d5eb8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.845649 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 655/1293] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:27.847539 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51144c02-4182-4a2a-8b0c-edba111d5eb8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:27.848774 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf4179f-2108-4ad3-8778-787dbafd7872 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.849067 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf4179f-2108-4ad3-8778-787dbafd7872 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.849188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51144c02-4182-4a2a-8b0c-edba111d5eb8 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:27.850166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf4179f-2108-4ad3-8778-787dbafd7872 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.865941 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 639/1294] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:27.867332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51144c02-4182-4a2a-8b0c-edba111d5eb8 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:27.879157 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51144c02-4182-4a2a-8b0c-edba111d5eb8 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:27.884685 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0e450f-9cd8-40be-8ec2-4dea1b33c519 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:27.886205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e450f-9cd8-40be-8ec2-4dea1b33c519 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.886205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e450f-9cd8-40be-8ec2-4dea1b33c519 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.886434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e450f-9cd8-40be-8ec2-4dea1b33c519 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.887242 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0e450f-9cd8-40be-8ec2-4dea1b33c519 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:27.891352 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0e450f-9cd8-40be-8ec2-4dea1b33c519 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:27.895636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-acf4179f-2108-4ad3-8778-787dbafd7872 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:27.896880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-acf4179f-2108-4ad3-8778-787dbafd7872 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:27.916796 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 656/1295] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:27.918837 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf4179f-2108-4ad3-8778-787dbafd7872 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:27.938064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 640/1296] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:56:27 2022] POST /identity/v3/projects => generated 403 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:27.940022 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906a6c4d-d9a9-4d5b-946f-920cad3228e7 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:27.940022 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906a6c4d-d9a9-4d5b-946f-920cad3228e7 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:27.940486 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906a6c4d-d9a9-4d5b-946f-920cad3228e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:27.950838 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf4179f-2108-4ad3-8778-787dbafd7872 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:27.969170 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-906a6c4d-d9a9-4d5b-946f-920cad3228e7 None None] MFA Rules not processed for user `c8cace029a92406f850bb0a4ca07c461`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:27.970431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-906a6c4d-d9a9-4d5b-946f-920cad3228e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:27.997266 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:28.074836 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588bb23f-32ef-47fe-a6a3-cadc7d96c157 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:28.075521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588bb23f-32ef-47fe-a6a3-cadc7d96c157 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:28.075831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588bb23f-32ef-47fe-a6a3-cadc7d96c157 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:28.076107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588bb23f-32ef-47fe-a6a3-cadc7d96c157 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:28.079697 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588bb23f-32ef-47fe-a6a3-cadc7d96c157 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:28.081726 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588bb23f-32ef-47fe-a6a3-cadc7d96c157 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:28.131979 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 657/1297] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:56:27 2022] POST /identity/v3/projects => generated 379 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:28.133843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588bb23f-32ef-47fe-a6a3-cadc7d96c157 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:28.155244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 641/1298] 10.222.0.36 () {58 vars in 1009 bytes} [Mon Oct 31 14:56:27 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:28.164974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906a6c4d-d9a9-4d5b-946f-920cad3228e7 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:28.278591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588bb23f-32ef-47fe-a6a3-cadc7d96c157 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:28.291395 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acd9e94-e8fd-4ff5-8b69-1cee4ac3c7ce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:28.292320 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acd9e94-e8fd-4ff5-8b69-1cee4ac3c7ce service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:28.292614 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acd9e94-e8fd-4ff5-8b69-1cee4ac3c7ce service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:28.292893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acd9e94-e8fd-4ff5-8b69-1cee4ac3c7ce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:28.302042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906a6c4d-d9a9-4d5b-946f-920cad3228e7 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:28.310724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0700d66-e60c-4cb3-82e1-221c4adc13fc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:28.313061 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0700d66-e60c-4cb3-82e1-221c4adc13fc service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:28.316072 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0700d66-e60c-4cb3-82e1-221c4adc13fc service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:28.316366 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0700d66-e60c-4cb3-82e1-221c4adc13fc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:28.454430 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acd9e94-e8fd-4ff5-8b69-1cee4ac3c7ce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:28.459329 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acd9e94-e8fd-4ff5-8b69-1cee4ac3c7ce service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:28.478615 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0700d66-e60c-4cb3-82e1-221c4adc13fc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:28.479495 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 658/1299] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:28.481507 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acd9e94-e8fd-4ff5-8b69-1cee4ac3c7ce service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:28.482673 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0700d66-e60c-4cb3-82e1-221c4adc13fc service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:28.491530 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:28.495187 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acd9e94-e8fd-4ff5-8b69-1cee4ac3c7ce service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:28.502234 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 642/1300] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:28.503910 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0700d66-e60c-4cb3-82e1-221c4adc13fc service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:28.505930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a279c0-64f4-4ec9-a3fa-20870ad3b148 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:28.506937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a279c0-64f4-4ec9-a3fa-20870ad3b148 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:28.507118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a279c0-64f4-4ec9-a3fa-20870ad3b148 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:28.507513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a279c0-64f4-4ec9-a3fa-20870ad3b148 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:28.508287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a279c0-64f4-4ec9-a3fa-20870ad3b148 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:28.510500 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a279c0-64f4-4ec9-a3fa-20870ad3b148 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:28.518557 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0700d66-e60c-4cb3-82e1-221c4adc13fc service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:28.524792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7685efae-d274-4645-aec7-82520362233d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:28.526014 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7685efae-d274-4645-aec7-82520362233d admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:28.526646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7685efae-d274-4645-aec7-82520362233d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:28.527684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7685efae-d274-4645-aec7-82520362233d admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:28.529356 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7685efae-d274-4645-aec7-82520362233d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:28.533321 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7685efae-d274-4645-aec7-82520362233d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:28.560693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 659/1301] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:56:28 2022] POST /identity/v3/users => generated 308 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:28.562049 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a279c0-64f4-4ec9-a3fa-20870ad3b148 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:28.588056 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 643/1302] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:56:28 2022] POST /identity/v3/users => generated 296 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:28.590863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7685efae-d274-4645-aec7-82520362233d admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:28.659429 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7685efae-d274-4645-aec7-82520362233d admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:28.659429 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e873b15c-9e4a-4f7b-8c69-3f8625e52810 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:28.659429 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e873b15c-9e4a-4f7b-8c69-3f8625e52810 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:28.659429 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e873b15c-9e4a-4f7b-8c69-3f8625e52810 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:28.660203 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e873b15c-9e4a-4f7b-8c69-3f8625e52810 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:28.660203 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e873b15c-9e4a-4f7b-8c69-3f8625e52810 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:28.660203 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e873b15c-9e4a-4f7b-8c69-3f8625e52810 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:28.697338 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:28.698195 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 644/1303] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:28 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:28.699324 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e873b15c-9e4a-4f7b-8c69-3f8625e52810 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:28.725126 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e873b15c-9e4a-4f7b-8c69-3f8625e52810 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:28.731871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51938cd1-1d11-404e-9ea0-eb2675a639dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:28.732804 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51938cd1-1d11-404e-9ea0-eb2675a639dd admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:28.733253 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51938cd1-1d11-404e-9ea0-eb2675a639dd admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:28.733774 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51938cd1-1d11-404e-9ea0-eb2675a639dd admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:28.734800 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51938cd1-1d11-404e-9ea0-eb2675a639dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:28.737223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51938cd1-1d11-404e-9ea0-eb2675a639dd admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:28.741644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a279c0-64f4-4ec9-a3fa-20870ad3b148 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:28.754104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60eefd28-8f68-4f5a-8d2d-c63afa8269ca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:28.754464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60eefd28-8f68-4f5a-8d2d-c63afa8269ca service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:28.754749 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60eefd28-8f68-4f5a-8d2d-c63afa8269ca service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:28.755027 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60eefd28-8f68-4f5a-8d2d-c63afa8269ca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:28.761910 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 645/1304] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:28 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:28.769057 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51938cd1-1d11-404e-9ea0-eb2675a639dd admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:28.785314 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51938cd1-1d11-404e-9ea0-eb2675a639dd admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:28.796991 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029b2c0a-5752-4282-9bd6-225c71712475 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:28.798064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029b2c0a-5752-4282-9bd6-225c71712475 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:28.798821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029b2c0a-5752-4282-9bd6-225c71712475 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:28.799429 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029b2c0a-5752-4282-9bd6-225c71712475 admin admin] PATH_INFO: `/v3/projects/febf47c5b6ed49c185dfd2b768614df6/users/665cc9fd63694dc98c96a267449a2345/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:28.819445 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029b2c0a-5752-4282-9bd6-225c71712475 admin admin] RBAC: Authorizing `identity:create_grant(project_id=febf47c5b6ed49c185dfd2b768614df6, user_id=665cc9fd63694dc98c96a267449a2345, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:28.824553 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029b2c0a-5752-4282-9bd6-225c71712475 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:28.847768 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 646/1305] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:28 2022] PUT /identity/v3/projects/febf47c5b6ed49c185dfd2b768614df6/users/665cc9fd63694dc98c96a267449a2345/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:28.852034 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029b2c0a-5752-4282-9bd6-225c71712475 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:28.908675 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60eefd28-8f68-4f5a-8d2d-c63afa8269ca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:28.911086 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60eefd28-8f68-4f5a-8d2d-c63afa8269ca service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:28.937428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 660/1306] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:28.939287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60eefd28-8f68-4f5a-8d2d-c63afa8269ca service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:28.970875 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60eefd28-8f68-4f5a-8d2d-c63afa8269ca service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:28.975271 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029b2c0a-5752-4282-9bd6-225c71712475 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:28.981362 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c545974a-13b0-4e1f-bb8f-71777b7d2b44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:28.982250 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c545974a-13b0-4e1f-bb8f-71777b7d2b44 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:28.982457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c545974a-13b0-4e1f-bb8f-71777b7d2b44 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:28.982857 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c545974a-13b0-4e1f-bb8f-71777b7d2b44 admin admin] PATH_INFO: `/v3/projects/5c2b1a5357ae42b9b12418dd405816c7/users/10f72295f40a4af9ab2219cc59b63ab6/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:28.995602 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e016b30b-f076-4d5b-813c-d70f7d0b5b0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:28.996241 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e016b30b-f076-4d5b-813c-d70f7d0b5b0e admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:28.996530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e016b30b-f076-4d5b-813c-d70f7d0b5b0e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:28.999321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e016b30b-f076-4d5b-813c-d70f7d0b5b0e admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:29.000098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e016b30b-f076-4d5b-813c-d70f7d0b5b0e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:29.006966 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e016b30b-f076-4d5b-813c-d70f7d0b5b0e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:29.008403 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c545974a-13b0-4e1f-bb8f-71777b7d2b44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c2b1a5357ae42b9b12418dd405816c7, user_id=10f72295f40a4af9ab2219cc59b63ab6, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:29.011353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c545974a-13b0-4e1f-bb8f-71777b7d2b44 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:29.036174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 647/1307] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:28 2022] PUT /identity/v3/projects/5c2b1a5357ae42b9b12418dd405816c7/users/10f72295f40a4af9ab2219cc59b63ab6/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:29.037324 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c545974a-13b0-4e1f-bb8f-71777b7d2b44 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:29.042326 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 661/1308] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:28 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:29.043527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e016b30b-f076-4d5b-813c-d70f7d0b5b0e admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:29.084100 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e016b30b-f076-4d5b-813c-d70f7d0b5b0e admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:29.176679 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85e149f-40be-40cb-a649-f8343038e360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:29.177541 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85e149f-40be-40cb-a649-f8343038e360 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:29.177867 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85e149f-40be-40cb-a649-f8343038e360 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:29.178146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85e149f-40be-40cb-a649-f8343038e360 admin admin] PATH_INFO: `/v3/projects/febf47c5b6ed49c185dfd2b768614df6/users/665cc9fd63694dc98c96a267449a2345/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:29.180237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85e149f-40be-40cb-a649-f8343038e360 admin admin] RBAC: Authorizing `identity:create_grant(project_id=febf47c5b6ed49c185dfd2b768614df6, user_id=665cc9fd63694dc98c96a267449a2345, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:29.180724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c545974a-13b0-4e1f-bb8f-71777b7d2b44 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:29.183275 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85e149f-40be-40cb-a649-f8343038e360 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:29.187766 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef8c531-ecb6-4bd8-b863-cee96eec98c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:29.188527 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef8c531-ecb6-4bd8-b863-cee96eec98c2 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:29.188915 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef8c531-ecb6-4bd8-b863-cee96eec98c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:29.189377 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef8c531-ecb6-4bd8-b863-cee96eec98c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:29.190240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef8c531-ecb6-4bd8-b863-cee96eec98c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:29.192910 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef8c531-ecb6-4bd8-b863-cee96eec98c2 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:29.208655 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 662/1309] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:29 2022] PUT /identity/v3/projects/febf47c5b6ed49c185dfd2b768614df6/users/665cc9fd63694dc98c96a267449a2345/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:29.210909 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71d8299-f267-4e38-b2f5-6f7993a16a8b None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:29.211508 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71d8299-f267-4e38-b2f5-6f7993a16a8b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:29.211755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 648/1310] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:29 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:29.211908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71d8299-f267-4e38-b2f5-6f7993a16a8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:29.212842 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef8c531-ecb6-4bd8-b863-cee96eec98c2 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:29.226514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef8c531-ecb6-4bd8-b863-cee96eec98c2 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:29.236872 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:29.245750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-a71d8299-f267-4e38-b2f5-6f7993a16a8b None None] MFA Rules not processed for user `665cc9fd63694dc98c96a267449a2345`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:29.247147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a71d8299-f267-4e38-b2f5-6f7993a16a8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:29.337370 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:29.357140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6350fb4-343d-495a-8cbb-c3f0ad71d7e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:29.358373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6350fb4-343d-495a-8cbb-c3f0ad71d7e5 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:29.358703 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6350fb4-343d-495a-8cbb-c3f0ad71d7e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:29.358983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6350fb4-343d-495a-8cbb-c3f0ad71d7e5 admin admin] PATH_INFO: `/v3/projects/5c2b1a5357ae42b9b12418dd405816c7/users/10f72295f40a4af9ab2219cc59b63ab6/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:29.362676 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6350fb4-343d-495a-8cbb-c3f0ad71d7e5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c2b1a5357ae42b9b12418dd405816c7, user_id=10f72295f40a4af9ab2219cc59b63ab6, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:29.366917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 663/1311] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:29 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:29.370667 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6350fb4-343d-495a-8cbb-c3f0ad71d7e5 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:29.382494 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71d8299-f267-4e38-b2f5-6f7993a16a8b None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:29.418924 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 649/1312] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:29 2022] PUT /identity/v3/projects/5c2b1a5357ae42b9b12418dd405816c7/users/10f72295f40a4af9ab2219cc59b63ab6/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:29.421706 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db35dbf4-23dd-4526-a3a0-dc637f1b153f None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:29.422021 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db35dbf4-23dd-4526-a3a0-dc637f1b153f None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:29.423846 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db35dbf4-23dd-4526-a3a0-dc637f1b153f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:29.491483 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-db35dbf4-23dd-4526-a3a0-dc637f1b153f None None] MFA Rules not processed for user `10f72295f40a4af9ab2219cc59b63ab6`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:29.495005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-db35dbf4-23dd-4526-a3a0-dc637f1b153f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:29.514874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71d8299-f267-4e38-b2f5-6f7993a16a8b None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:29.655643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ce2b65-6864-4164-b52d-bba717566172 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:29.656202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ce2b65-6864-4164-b52d-bba717566172 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:29.656488 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ce2b65-6864-4164-b52d-bba717566172 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:29.657042 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ce2b65-6864-4164-b52d-bba717566172 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:29.687432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 650/1313] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:29 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:29.782331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ce2b65-6864-4164-b52d-bba717566172 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:29.784005 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ce2b65-6864-4164-b52d-bba717566172 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:29.814733 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 664/1314] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:29 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:29.953704 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db35dbf4-23dd-4526-a3a0-dc637f1b153f None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:29.969867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db35dbf4-23dd-4526-a3a0-dc637f1b153f None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:29.980270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a71dfc-5e35-4d97-a056-d4a0e6e64055 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:29.981531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a71dfc-5e35-4d97-a056-d4a0e6e64055 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:29.983213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a71dfc-5e35-4d97-a056-d4a0e6e64055 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:29.986073 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a71dfc-5e35-4d97-a056-d4a0e6e64055 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:30.015945 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a71dfc-5e35-4d97-a056-d4a0e6e64055 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:30.017502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a71dfc-5e35-4d97-a056-d4a0e6e64055 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:30.046923 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 651/1315] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:29 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:30.367539 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ce2b65-6864-4164-b52d-bba717566172 service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:30.392409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ce2b65-6864-4164-b52d-bba717566172 service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:30.396613 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3914e4a-7934-497b-ba1b-2b08bf9d4df0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:30.397969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3914e4a-7934-497b-ba1b-2b08bf9d4df0 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:30.398436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3914e4a-7934-497b-ba1b-2b08bf9d4df0 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:30.398885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3914e4a-7934-497b-ba1b-2b08bf9d4df0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:30.448110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a71dfc-5e35-4d97-a056-d4a0e6e64055 service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:30.492424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3914e4a-7934-497b-ba1b-2b08bf9d4df0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:30.493962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3914e4a-7934-497b-ba1b-2b08bf9d4df0 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:30.512154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:30.534965 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 665/1316] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:30 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:30.622640 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a71dfc-5e35-4d97-a056-d4a0e6e64055 service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:30.649847 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2869c2b3-61a2-41d8-9e2e-9807b0a3c29f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:30.651456 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2869c2b3-61a2-41d8-9e2e-9807b0a3c29f admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:30.651994 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2869c2b3-61a2-41d8-9e2e-9807b0a3c29f admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:30.652633 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2869c2b3-61a2-41d8-9e2e-9807b0a3c29f admin admin] PATH_INFO: `/v3/users/a38d8c2788cf40249c1daedc0ad4cdf6` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:30.654967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2869c2b3-61a2-41d8-9e2e-9807b0a3c29f admin admin] RBAC: Authorizing `identity:delete_user(user_id=a38d8c2788cf40249c1daedc0ad4cdf6)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:30.660330 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: WARNING py.warnings [None req-2869c2b3-61a2-41d8-9e2e-9807b0a3c29f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 31 14:56:30.660330 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: warnings.warn(msg) Oct 31 14:56:30.660330 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: Oct 31 14:56:30.661854 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2869c2b3-61a2-41d8-9e2e-9807b0a3c29f admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:30.764370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:30.831005 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085c6952-7c7f-4086-8a7e-01dc5e3f988a None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:30.833123 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085c6952-7c7f-4086-8a7e-01dc5e3f988a None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:30.833583 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085c6952-7c7f-4086-8a7e-01dc5e3f988a None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:30.836534 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 666/1317] 10.222.0.36 () {54 vars in 961 bytes} [Mon Oct 31 14:56:30 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:56:30.837587 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2869c2b3-61a2-41d8-9e2e-9807b0a3c29f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a38d8c2788cf40249c1daedc0ad4cdf6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:30.847108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f8628e-4283-468b-a831-42715cfe0c0e None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:30.848040 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f8628e-4283-468b-a831-42715cfe0c0e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:30.848970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f8628e-4283-468b-a831-42715cfe0c0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:30.880395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2869c2b3-61a2-41d8-9e2e-9807b0a3c29f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a38d8c2788cf40249c1daedc0ad4cdf6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:30.881255 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2869c2b3-61a2-41d8-9e2e-9807b0a3c29f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a38d8c2788cf40249c1daedc0ad4cdf6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:30.882741 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2869c2b3-61a2-41d8-9e2e-9807b0a3c29f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a38d8c2788cf40249c1daedc0ad4cdf6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:30.895373 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-b6f8628e-4283-468b-a831-42715cfe0c0e None None] MFA Rules not processed for user `c8cace029a92406f850bb0a4ca07c461`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:30.901401 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b6f8628e-4283-468b-a831-42715cfe0c0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:30.947439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2869c2b3-61a2-41d8-9e2e-9807b0a3c29f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a38d8c2788cf40249c1daedc0ad4cdf6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:30.982218 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 652/1318] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:56:30 2022] DELETE /identity/v3/users/a38d8c2788cf40249c1daedc0ad4cdf6 => generated 0 bytes in 535 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:30.989679 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f5b39d-c776-4dd0-bb7d-80974ca22a09 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:30.989975 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f5b39d-c776-4dd0-bb7d-80974ca22a09 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:30.990252 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f5b39d-c776-4dd0-bb7d-80974ca22a09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:31.056775 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-97f5b39d-c776-4dd0-bb7d-80974ca22a09 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:31.058474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-97f5b39d-c776-4dd0-bb7d-80974ca22a09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:31.145971 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 667/1319] 10.222.0.36 () {58 vars in 1058 bytes} [Mon Oct 31 14:56:30 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:56:31.151116 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6655896-d3a4-4c02-b52b-b7f4f7aa14d2 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:31.156418 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6655896-d3a4-4c02-b52b-b7f4f7aa14d2 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:31.158049 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6655896-d3a4-4c02-b52b-b7f4f7aa14d2 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:31.161881 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 668/1320] 10.222.0.36 () {54 vars in 912 bytes} [Mon Oct 31 14:56:31 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:56:31.170089 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b70aa4-3062-423b-8786-3d7aef000596 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:31.171094 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b70aa4-3062-423b-8786-3d7aef000596 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:31.171521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b70aa4-3062-423b-8786-3d7aef000596 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:31.194123 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 653/1321] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:31.210261 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-01b70aa4-3062-423b-8786-3d7aef000596 None None] MFA Rules not processed for user `c8cace029a92406f850bb0a4ca07c461`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:31.211680 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-01b70aa4-3062-423b-8786-3d7aef000596 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:31.238463 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f5b39d-c776-4dd0-bb7d-80974ca22a09 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:31.240367 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-97f5b39d-c776-4dd0-bb7d-80974ca22a09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:31.387218 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f5b39d-c776-4dd0-bb7d-80974ca22a09 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:31.397198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 669/1322] 10.222.0.36 () {58 vars in 1009 bytes} [Mon Oct 31 14:56:31 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:31.397780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2c8b12-1e9e-40f8-85f1-5867a682223e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:31.398393 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2c8b12-1e9e-40f8-85f1-5867a682223e service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:31.398671 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2c8b12-1e9e-40f8-85f1-5867a682223e service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:31.399225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2c8b12-1e9e-40f8-85f1-5867a682223e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:31.405571 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b70aa4-3062-423b-8786-3d7aef000596 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:31.408633 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-01b70aa4-3062-423b-8786-3d7aef000596 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:31.423283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2c8b12-1e9e-40f8-85f1-5867a682223e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:31.424932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2c8b12-1e9e-40f8-85f1-5867a682223e service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:31.448005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 654/1323] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:31.529159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b70aa4-3062-423b-8786-3d7aef000596 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:31.534428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb4e12e-671f-4b16-9b11-1f005afbb229 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:31.535599 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb4e12e-671f-4b16-9b11-1f005afbb229 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:31.536165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb4e12e-671f-4b16-9b11-1f005afbb229 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:31.536660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb4e12e-671f-4b16-9b11-1f005afbb229 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:31.551055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb4e12e-671f-4b16-9b11-1f005afbb229 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:31.568555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb4e12e-671f-4b16-9b11-1f005afbb229 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:31.587090 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 670/1324] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:31 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:56:33.327515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2c8b12-1e9e-40f8-85f1-5867a682223e service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:33.329254 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-7b2c8b12-1e9e-40f8-85f1-5867a682223e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:33.359798 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2c8b12-1e9e-40f8-85f1-5867a682223e service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:33.391126 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f01bede-7b2f-4ed3-9d65-78721da48198 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:33.392056 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f01bede-7b2f-4ed3-9d65-78721da48198 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:33.392551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f01bede-7b2f-4ed3-9d65-78721da48198 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:33.393036 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f01bede-7b2f-4ed3-9d65-78721da48198 admin admin] PATH_INFO: `/v3/users/78dbaf070ba94a0894cecc104ab26e42` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:33.398720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f01bede-7b2f-4ed3-9d65-78721da48198 admin admin] RBAC: Authorizing `identity:delete_user(user_id=78dbaf070ba94a0894cecc104ab26e42)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:33.401026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f01bede-7b2f-4ed3-9d65-78721da48198 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:33.501561 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:33.517914 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2f01bede-7b2f-4ed3-9d65-78721da48198 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '78dbaf070ba94a0894cecc104ab26e42'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:33.535996 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2f01bede-7b2f-4ed3-9d65-78721da48198 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '78dbaf070ba94a0894cecc104ab26e42'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:33.536967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2f01bede-7b2f-4ed3-9d65-78721da48198 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '78dbaf070ba94a0894cecc104ab26e42'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:33.537902 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2f01bede-7b2f-4ed3-9d65-78721da48198 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '78dbaf070ba94a0894cecc104ab26e42'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:33.567729 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2f01bede-7b2f-4ed3-9d65-78721da48198 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '78dbaf070ba94a0894cecc104ab26e42'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:33.590677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 655/1325] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:56:33 2022] DELETE /identity/v3/users/78dbaf070ba94a0894cecc104ab26e42 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:33.596246 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839bfc74-2ff5-40db-bee2-b12e04601115 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:33.596808 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839bfc74-2ff5-40db-bee2-b12e04601115 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:33.596895 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839bfc74-2ff5-40db-bee2-b12e04601115 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:33.633856 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-839bfc74-2ff5-40db-bee2-b12e04601115 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:33.638355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-839bfc74-2ff5-40db-bee2-b12e04601115 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:33.699786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:33.807882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 671/1326] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:33.898316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f01bede-7b2f-4ed3-9d65-78721da48198 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:33.899861 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2f01bede-7b2f-4ed3-9d65-78721da48198 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:34.010689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f01bede-7b2f-4ed3-9d65-78721da48198 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:34.016473 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a4c636-3464-474a-a2e4-59fa5135a007 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:34.017162 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a4c636-3464-474a-a2e4-59fa5135a007 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:34.017447 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a4c636-3464-474a-a2e4-59fa5135a007 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:34.017724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a4c636-3464-474a-a2e4-59fa5135a007 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:34.031622 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a4c636-3464-474a-a2e4-59fa5135a007 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:34.037531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a4c636-3464-474a-a2e4-59fa5135a007 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:34.049707 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 656/1327] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:34.964050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839bfc74-2ff5-40db-bee2-b12e04601115 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:34.966446 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-839bfc74-2ff5-40db-bee2-b12e04601115 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:35.011207 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839bfc74-2ff5-40db-bee2-b12e04601115 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:35.046641 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fa6a50-866e-493f-954a-0c000d30fa7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:35.047386 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fa6a50-866e-493f-954a-0c000d30fa7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:35.049021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fa6a50-866e-493f-954a-0c000d30fa7e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:35.049508 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fa6a50-866e-493f-954a-0c000d30fa7e admin admin] PATH_INFO: `/v3/projects/d8dc1ffa3cab4805a4b7010ee4a5bc19` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:35.054139 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fa6a50-866e-493f-954a-0c000d30fa7e admin admin] RBAC: Authorizing `identity:delete_project(project_id=d8dc1ffa3cab4805a4b7010ee4a5bc19)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:35.055510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: WARNING py.warnings [None req-85fa6a50-866e-493f-954a-0c000d30fa7e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 31 14:56:35.055510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: warnings.warn(msg) Oct 31 14:56:35.055510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: Oct 31 14:56:35.059436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fa6a50-866e-493f-954a-0c000d30fa7e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:35.233589 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-85fa6a50-866e-493f-954a-0c000d30fa7e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd8dc1ffa3cab4805a4b7010ee4a5bc19', 'request_id': 'req-85fa6a50-866e-493f-954a-0c000d30fa7e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:35.260525 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-85fa6a50-866e-493f-954a-0c000d30fa7e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd8dc1ffa3cab4805a4b7010ee4a5bc19', 'request_id': 'req-85fa6a50-866e-493f-954a-0c000d30fa7e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:35.307367 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-85fa6a50-866e-493f-954a-0c000d30fa7e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd8dc1ffa3cab4805a4b7010ee4a5bc19', 'request_id': 'req-85fa6a50-866e-493f-954a-0c000d30fa7e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:35.382470 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-85fa6a50-866e-493f-954a-0c000d30fa7e admin admin] The token cache is being invalidate because project d8dc1ffa3cab4805a4b7010ee4a5bc19 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:56:35.383059 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-85fa6a50-866e-493f-954a-0c000d30fa7e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:35.384933 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-85fa6a50-866e-493f-954a-0c000d30fa7e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:35.390261 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 672/1328] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:56:34 2022] DELETE /identity/v3/projects/d8dc1ffa3cab4805a4b7010ee4a5bc19 => generated 0 bytes in 426 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:35.394439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a4c636-3464-474a-a2e4-59fa5135a007 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:35.412760 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a4c636-3464-474a-a2e4-59fa5135a007 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:35.603629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f309028-4c6b-46fc-9727-36ac72290900 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:35.605407 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f309028-4c6b-46fc-9727-36ac72290900 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:35.606531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f309028-4c6b-46fc-9727-36ac72290900 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:35.606913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f309028-4c6b-46fc-9727-36ac72290900 admin admin] PATH_INFO: `/v3/users/665cc9fd63694dc98c96a267449a2345` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:35.608943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f309028-4c6b-46fc-9727-36ac72290900 admin admin] RBAC: Authorizing `identity:delete_user(user_id=665cc9fd63694dc98c96a267449a2345)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:35.611614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f309028-4c6b-46fc-9727-36ac72290900 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:35.735440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-9f309028-4c6b-46fc-9727-36ac72290900 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '665cc9fd63694dc98c96a267449a2345'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:35.756156 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-9f309028-4c6b-46fc-9727-36ac72290900 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '665cc9fd63694dc98c96a267449a2345'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:35.757126 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-9f309028-4c6b-46fc-9727-36ac72290900 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '665cc9fd63694dc98c96a267449a2345'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:35.757948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-9f309028-4c6b-46fc-9727-36ac72290900 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '665cc9fd63694dc98c96a267449a2345'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:35.760564 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:35.775062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-9f309028-4c6b-46fc-9727-36ac72290900 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '665cc9fd63694dc98c96a267449a2345'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:35.803880 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 657/1329] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:56:35 2022] DELETE /identity/v3/users/665cc9fd63694dc98c96a267449a2345 => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:36.471015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fa6a50-866e-493f-954a-0c000d30fa7e admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:36.473327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-85fa6a50-866e-493f-954a-0c000d30fa7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:36.506349 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fa6a50-866e-493f-954a-0c000d30fa7e admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:36.628285 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5e7776-7ab4-4633-9a17-53df5799cd45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:36.629197 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5e7776-7ab4-4633-9a17-53df5799cd45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:36.629908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5e7776-7ab4-4633-9a17-53df5799cd45 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:36.630336 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5e7776-7ab4-4633-9a17-53df5799cd45 admin admin] PATH_INFO: `/v3/projects/febf47c5b6ed49c185dfd2b768614df6` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:36.636483 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5e7776-7ab4-4633-9a17-53df5799cd45 admin admin] RBAC: Authorizing `identity:delete_project(project_id=febf47c5b6ed49c185dfd2b768614df6)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:36.639266 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5e7776-7ab4-4633-9a17-53df5799cd45 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:36.749988 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5b5e7776-7ab4-4633-9a17-53df5799cd45 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'febf47c5b6ed49c185dfd2b768614df6', 'request_id': 'req-5b5e7776-7ab4-4633-9a17-53df5799cd45'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:36.751837 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:36.758839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5b5e7776-7ab4-4633-9a17-53df5799cd45 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'febf47c5b6ed49c185dfd2b768614df6', 'request_id': 'req-5b5e7776-7ab4-4633-9a17-53df5799cd45'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:36.770323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5b5e7776-7ab4-4633-9a17-53df5799cd45 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'febf47c5b6ed49c185dfd2b768614df6', 'request_id': 'req-5b5e7776-7ab4-4633-9a17-53df5799cd45'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:36.814733 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5b5e7776-7ab4-4633-9a17-53df5799cd45 admin admin] The token cache is being invalidate because project febf47c5b6ed49c185dfd2b768614df6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:56:36.815221 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5b5e7776-7ab4-4633-9a17-53df5799cd45 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:36.816221 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5b5e7776-7ab4-4633-9a17-53df5799cd45 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:36.824747 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e48885c-fffb-49e3-b8a0-133f1f5327b7 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:36.825162 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e48885c-fffb-49e3-b8a0-133f1f5327b7 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:36.825339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e48885c-fffb-49e3-b8a0-133f1f5327b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:36.837052 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 673/1330] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:56:36 2022] DELETE /identity/v3/projects/febf47c5b6ed49c185dfd2b768614df6 => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:36.907420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-8e48885c-fffb-49e3-b8a0-133f1f5327b7 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:36.908733 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-8e48885c-fffb-49e3-b8a0-133f1f5327b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:37.050278 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 658/1331] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:37.063146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60406b6e-d3cc-46c3-aaa7-8596ae9a1b1f None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:37.065372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60406b6e-d3cc-46c3-aaa7-8596ae9a1b1f None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:37.065372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60406b6e-d3cc-46c3-aaa7-8596ae9a1b1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:37.101057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-60406b6e-d3cc-46c3-aaa7-8596ae9a1b1f None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:37.102439 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-60406b6e-d3cc-46c3-aaa7-8596ae9a1b1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:37.122468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 674/1332] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:37.124182 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e48885c-fffb-49e3-b8a0-133f1f5327b7 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:37.133786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e48885c-fffb-49e3-b8a0-133f1f5327b7 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:37.143728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91845cd6-3857-4082-803b-e3cbade25477 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:37.144572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91845cd6-3857-4082-803b-e3cbade25477 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:37.144921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91845cd6-3857-4082-803b-e3cbade25477 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:37.145263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91845cd6-3857-4082-803b-e3cbade25477 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:37.145993 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91845cd6-3857-4082-803b-e3cbade25477 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:56:37.146366 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91845cd6-3857-4082-803b-e3cbade25477 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:37.149646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91845cd6-3857-4082-803b-e3cbade25477 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:37.186704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60406b6e-d3cc-46c3-aaa7-8596ae9a1b1f None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:37.197690 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 659/1333] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:56:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:56:37.199739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60406b6e-d3cc-46c3-aaa7-8596ae9a1b1f None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:37.215106 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e970bf5b-366c-4ec9-9049-f811ba9b23f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:37.215875 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e970bf5b-366c-4ec9-9049-f811ba9b23f7 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:37.216160 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e970bf5b-366c-4ec9-9049-f811ba9b23f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:37.217002 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e970bf5b-366c-4ec9-9049-f811ba9b23f7 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:37.218196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e970bf5b-366c-4ec9-9049-f811ba9b23f7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:37.221403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e970bf5b-366c-4ec9-9049-f811ba9b23f7 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:37.254906 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3349d1-bd73-4932-9590-43fee34f3d85 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:37.256091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3349d1-bd73-4932-9590-43fee34f3d85 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:37.256629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3349d1-bd73-4932-9590-43fee34f3d85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:37.280828 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 675/1334] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:56:37 2022] POST /identity/v3/projects => generated 371 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:37.283275 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e970bf5b-366c-4ec9-9049-f811ba9b23f7 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:37.360445 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-ff3349d1-bd73-4932-9590-43fee34f3d85 None None] MFA Rules not processed for user `77af489d5daa4d028b737c5f4efc93db`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:37.361754 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ff3349d1-bd73-4932-9590-43fee34f3d85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:37.392633 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e970bf5b-366c-4ec9-9049-f811ba9b23f7 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:37.399938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f952f810-d849-4189-bfcb-3177bdb5a99a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:37.401426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f952f810-d849-4189-bfcb-3177bdb5a99a admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:37.402053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f952f810-d849-4189-bfcb-3177bdb5a99a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:37.402563 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f952f810-d849-4189-bfcb-3177bdb5a99a admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:37.403442 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f952f810-d849-4189-bfcb-3177bdb5a99a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:37.406500 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f952f810-d849-4189-bfcb-3177bdb5a99a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:37.458887 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 676/1335] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:56:37 2022] POST /identity/v3/users => generated 286 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:37.478486 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f952f810-d849-4189-bfcb-3177bdb5a99a admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:37.519640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f952f810-d849-4189-bfcb-3177bdb5a99a admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:37.542023 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42cf04d-73d0-4e33-9edf-e30ab19fa894 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:37.542678 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42cf04d-73d0-4e33-9edf-e30ab19fa894 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:37.542965 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42cf04d-73d0-4e33-9edf-e30ab19fa894 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:37.543241 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42cf04d-73d0-4e33-9edf-e30ab19fa894 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:37.549913 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42cf04d-73d0-4e33-9edf-e30ab19fa894 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:37.562046 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42cf04d-73d0-4e33-9edf-e30ab19fa894 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:37.562472 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 660/1336] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:37 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:37.564359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb09f42-1cee-40df-a52b-4cc6c81037d1 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:37.564926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb09f42-1cee-40df-a52b-4cc6c81037d1 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:37.565546 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb09f42-1cee-40df-a52b-4cc6c81037d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:37.572237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 677/1337] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:37 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:37.581058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42cf04d-73d0-4e33-9edf-e30ab19fa894 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:37.600017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42cf04d-73d0-4e33-9edf-e30ab19fa894 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:37.604724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-dcb09f42-1cee-40df-a52b-4cc6c81037d1 None None] MFA Rules not processed for user `77af489d5daa4d028b737c5f4efc93db`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:37.606546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4cc13a-0aac-45b5-abe5-3c593f5751ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:37.608100 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4cc13a-0aac-45b5-abe5-3c593f5751ae admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:37.608303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-dcb09f42-1cee-40df-a52b-4cc6c81037d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:37.608723 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4cc13a-0aac-45b5-abe5-3c593f5751ae admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:37.609142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4cc13a-0aac-45b5-abe5-3c593f5751ae admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:37.613096 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4cc13a-0aac-45b5-abe5-3c593f5751ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:37.616192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4cc13a-0aac-45b5-abe5-3c593f5751ae admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:37.629385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 661/1338] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:37 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:37.644641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb09f42-1cee-40df-a52b-4cc6c81037d1 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:37.645724 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 678/1339] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:37 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:37.646999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-dcb09f42-1cee-40df-a52b-4cc6c81037d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:37.652657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4cc13a-0aac-45b5-abe5-3c593f5751ae admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:37.685774 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4cc13a-0aac-45b5-abe5-3c593f5751ae admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:37.704196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404ef288-4d26-4b5c-8139-506b01914604 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:37.705157 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404ef288-4d26-4b5c-8139-506b01914604 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:37.705876 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404ef288-4d26-4b5c-8139-506b01914604 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:37.706379 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404ef288-4d26-4b5c-8139-506b01914604 admin admin] PATH_INFO: `/v3/projects/db9844d550444cf68b91c651487897fe/users/e01d58067e2546b9bf8502e7fa729154/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:37.735606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404ef288-4d26-4b5c-8139-506b01914604 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db9844d550444cf68b91c651487897fe, user_id=e01d58067e2546b9bf8502e7fa729154, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:37.742108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404ef288-4d26-4b5c-8139-506b01914604 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:37.761233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=96,keystone:DELETE=3,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:37.774834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 679/1340] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:37 2022] PUT /identity/v3/projects/db9844d550444cf68b91c651487897fe/users/e01d58067e2546b9bf8502e7fa729154/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:37.786587 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e3d6a7-9495-4e6d-8ac6-9d0ac35c32d6 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:37.787067 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e3d6a7-9495-4e6d-8ac6-9d0ac35c32d6 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:37.787197 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e3d6a7-9495-4e6d-8ac6-9d0ac35c32d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:37.809692 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb09f42-1cee-40df-a52b-4cc6c81037d1 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:37.883642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-e1e3d6a7-9495-4e6d-8ac6-9d0ac35c32d6 None None] MFA Rules not processed for user `e01d58067e2546b9bf8502e7fa729154`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:37.884952 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e1e3d6a7-9495-4e6d-8ac6-9d0ac35c32d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:37.916816 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02a7604-8f39-4a58-b47b-0d3a42c15237 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:37.924904 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02a7604-8f39-4a58-b47b-0d3a42c15237 service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:37.926360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02a7604-8f39-4a58-b47b-0d3a42c15237 service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:37.926815 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02a7604-8f39-4a58-b47b-0d3a42c15237 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:38.012609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02a7604-8f39-4a58-b47b-0d3a42c15237 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:38.017650 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02a7604-8f39-4a58-b47b-0d3a42c15237 service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:38.033260 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:DELETE=3,keystone:SELECT=91,keystone:INSERT=6 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:38.063867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 662/1341] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:37 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:38.100947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 680/1342] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:37 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 315 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:38.102071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02a7604-8f39-4a58-b47b-0d3a42c15237 service nova] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:38.196402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e3d6a7-9495-4e6d-8ac6-9d0ac35c32d6 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:38.197864 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e1e3d6a7-9495-4e6d-8ac6-9d0ac35c32d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:38.374236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02a7604-8f39-4a58-b47b-0d3a42c15237 service nova] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:38.401235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76830c51-bbce-4858-bdee-766b06e21878 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:38.403171 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76830c51-bbce-4858-bdee-766b06e21878 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:38.403405 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76830c51-bbce-4858-bdee-766b06e21878 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:38.407488 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76830c51-bbce-4858-bdee-766b06e21878 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:38.410558 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76830c51-bbce-4858-bdee-766b06e21878 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:38.413041 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76830c51-bbce-4858-bdee-766b06e21878 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:38.416103 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e3d6a7-9495-4e6d-8ac6-9d0ac35c32d6 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:38.435262 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07cff968-0f3b-4242-8df9-8a1a893e427f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:38.436159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cff968-0f3b-4242-8df9-8a1a893e427f admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:38.436439 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cff968-0f3b-4242-8df9-8a1a893e427f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:38.436861 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cff968-0f3b-4242-8df9-8a1a893e427f admin admin] PATH_INFO: `/v3/projects/713e15a5407049b2a8efd3ec38cd26cf` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:38.443972 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07cff968-0f3b-4242-8df9-8a1a893e427f admin admin] RBAC: Authorizing `identity:delete_project(project_id=713e15a5407049b2a8efd3ec38cd26cf)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:38.456118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07cff968-0f3b-4242-8df9-8a1a893e427f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:38.526793 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 663/1343] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:56:38 2022] POST /identity/v3/projects => generated 373 bytes in 425 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:38.534706 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76830c51-bbce-4858-bdee-766b06e21878 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:38.565929 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:38.636350 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-07cff968-0f3b-4242-8df9-8a1a893e427f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '713e15a5407049b2a8efd3ec38cd26cf', 'request_id': 'req-07cff968-0f3b-4242-8df9-8a1a893e427f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:38.665540 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-07cff968-0f3b-4242-8df9-8a1a893e427f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '713e15a5407049b2a8efd3ec38cd26cf', 'request_id': 'req-07cff968-0f3b-4242-8df9-8a1a893e427f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:38.679020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-07cff968-0f3b-4242-8df9-8a1a893e427f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '713e15a5407049b2a8efd3ec38cd26cf', 'request_id': 'req-07cff968-0f3b-4242-8df9-8a1a893e427f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:38.699639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76830c51-bbce-4858-bdee-766b06e21878 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:38.707427 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-07cff968-0f3b-4242-8df9-8a1a893e427f admin admin] The token cache is being invalidate because project 713e15a5407049b2a8efd3ec38cd26cf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:56:38.707966 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-07cff968-0f3b-4242-8df9-8a1a893e427f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:38.709087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-07cff968-0f3b-4242-8df9-8a1a893e427f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:38.713146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 681/1344] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:56:38 2022] DELETE /identity/v3/projects/713e15a5407049b2a8efd3ec38cd26cf => generated 0 bytes in 517 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:56:38.718656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1f0f44-9ad4-4684-a8d7-b60994efd37c None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:38.719125 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1f0f44-9ad4-4684-a8d7-b60994efd37c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:38.719595 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1f0f44-9ad4-4684-a8d7-b60994efd37c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:38.768232 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-5b1f0f44-9ad4-4684-a8d7-b60994efd37c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:38.769366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5b1f0f44-9ad4-4684-a8d7-b60994efd37c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:38.837421 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0ccc5c-a3d8-4990-b17f-891307ba41ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:38.838257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0ccc5c-a3d8-4990-b17f-891307ba41ef admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:38.844507 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0ccc5c-a3d8-4990-b17f-891307ba41ef admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:38.844795 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0ccc5c-a3d8-4990-b17f-891307ba41ef admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:38.845491 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0ccc5c-a3d8-4990-b17f-891307ba41ef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:38.847514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0ccc5c-a3d8-4990-b17f-891307ba41ef admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:38.913411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 664/1345] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:56:38 2022] POST /identity/v3/users => generated 293 bytes in 379 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:38.914786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0ccc5c-a3d8-4990-b17f-891307ba41ef admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:38.921587 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2b0ccc5c-a3d8-4990-b17f-891307ba41ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:38.945778 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 682/1346] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:38.947767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d889f6-fefa-46e2-a544-e1debd6c13f3 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:38.948163 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d889f6-fefa-46e2-a544-e1debd6c13f3 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:38.948540 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d889f6-fefa-46e2-a544-e1debd6c13f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:38.960756 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0ccc5c-a3d8-4990-b17f-891307ba41ef admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:38.975836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43599a2d-902c-49d1-add0-2fe7e75773d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:38.976464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43599a2d-902c-49d1-add0-2fe7e75773d9 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:38.976746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43599a2d-902c-49d1-add0-2fe7e75773d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:38.977021 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43599a2d-902c-49d1-add0-2fe7e75773d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:38.977675 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43599a2d-902c-49d1-add0-2fe7e75773d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:38.980095 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43599a2d-902c-49d1-add0-2fe7e75773d9 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:39.006604 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 665/1347] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:38 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:39.009109 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43599a2d-902c-49d1-add0-2fe7e75773d9 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:39.033215 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43599a2d-902c-49d1-add0-2fe7e75773d9 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:39.044319 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-36d889f6-fefa-46e2-a544-e1debd6c13f3 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:39.049933 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-36d889f6-fefa-46e2-a544-e1debd6c13f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:39.050330 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b9ac06-bbc8-4422-873e-66ca0ebb7ec9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:39.051318 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b9ac06-bbc8-4422-873e-66ca0ebb7ec9 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:39.051684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b9ac06-bbc8-4422-873e-66ca0ebb7ec9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:39.051965 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b9ac06-bbc8-4422-873e-66ca0ebb7ec9 admin admin] PATH_INFO: `/v3/projects/bb6becc00b6f414e92d6415a4df501e7/users/c779838194b9469396a6bf0b3a502dd0/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:39.079838 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 683/1348] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:39.082226 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d889f6-fefa-46e2-a544-e1debd6c13f3 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:39.106092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d889f6-fefa-46e2-a544-e1debd6c13f3 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:39.113683 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b9ac06-bbc8-4422-873e-66ca0ebb7ec9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb6becc00b6f414e92d6415a4df501e7, user_id=c779838194b9469396a6bf0b3a502dd0, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:39.121797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b9ac06-bbc8-4422-873e-66ca0ebb7ec9 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:39.127382 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fc77ed-e0fa-4153-8530-694ddda062d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:39.129769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fc77ed-e0fa-4153-8530-694ddda062d7 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:39.131961 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fc77ed-e0fa-4153-8530-694ddda062d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:39.132628 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fc77ed-e0fa-4153-8530-694ddda062d7 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:39.133866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fc77ed-e0fa-4153-8530-694ddda062d7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:56:39.134848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fc77ed-e0fa-4153-8530-694ddda062d7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:39.140207 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fc77ed-e0fa-4153-8530-694ddda062d7 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:39.145860 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:39.151674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 666/1349] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:39 2022] PUT /identity/v3/projects/bb6becc00b6f414e92d6415a4df501e7/users/c779838194b9469396a6bf0b3a502dd0/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:39.155775 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b9ac06-bbc8-4422-873e-66ca0ebb7ec9 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:39.182425 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 684/1350] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:56:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:56:39.183313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fc77ed-e0fa-4153-8530-694ddda062d7 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:39.223861 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fc77ed-e0fa-4153-8530-694ddda062d7 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:39.317812 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b9ac06-bbc8-4422-873e-66ca0ebb7ec9 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:39.348468 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9a160c-63f9-4a53-8750-442a8c97259a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:39.349589 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9a160c-63f9-4a53-8750-442a8c97259a admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:39.351038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9a160c-63f9-4a53-8750-442a8c97259a admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:39.351335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9a160c-63f9-4a53-8750-442a8c97259a admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:39.353190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9a160c-63f9-4a53-8750-442a8c97259a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:39.354855 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9a160c-63f9-4a53-8750-442a8c97259a admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:39.382116 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98882ec-75d2-469f-b726-978e1b7d1343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:39.388683 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98882ec-75d2-469f-b726-978e1b7d1343 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:39.389195 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98882ec-75d2-469f-b726-978e1b7d1343 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:39.389312 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 667/1351] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:39 2022] GET /identity/v3/roles => generated 1420 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:39.389705 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98882ec-75d2-469f-b726-978e1b7d1343 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:39.390540 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98882ec-75d2-469f-b726-978e1b7d1343 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:39.392854 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98882ec-75d2-469f-b726-978e1b7d1343 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:39.402775 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9a160c-63f9-4a53-8750-442a8c97259a admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:39.423739 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:39.437432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9a160c-63f9-4a53-8750-442a8c97259a admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:39.451428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 685/1352] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:56:39 2022] POST /identity/v3/projects => generated 363 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:39.452740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98882ec-75d2-469f-b726-978e1b7d1343 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:39.454441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1f2b4a-1ae4-40c7-a6c9-679b285a0ffc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:39.455139 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1f2b4a-1ae4-40c7-a6c9-679b285a0ffc admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:39.456529 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1f2b4a-1ae4-40c7-a6c9-679b285a0ffc admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:39.456816 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1f2b4a-1ae4-40c7-a6c9-679b285a0ffc admin admin] PATH_INFO: `/v3/projects/bb6becc00b6f414e92d6415a4df501e7/users/c779838194b9469396a6bf0b3a502dd0/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:39.464924 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1f2b4a-1ae4-40c7-a6c9-679b285a0ffc admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb6becc00b6f414e92d6415a4df501e7, user_id=c779838194b9469396a6bf0b3a502dd0, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:39.469263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1f2b4a-1ae4-40c7-a6c9-679b285a0ffc admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:39.504610 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 668/1353] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:39 2022] PUT /identity/v3/projects/bb6becc00b6f414e92d6415a4df501e7/users/c779838194b9469396a6bf0b3a502dd0/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:39.512276 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ae3972-a0fd-43f8-8c7f-891bceecfc2b None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:39.512561 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ae3972-a0fd-43f8-8c7f-891bceecfc2b None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:39.512840 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ae3972-a0fd-43f8-8c7f-891bceecfc2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:39.573450 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-68ae3972-a0fd-43f8-8c7f-891bceecfc2b None None] MFA Rules not processed for user `c779838194b9469396a6bf0b3a502dd0`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:39.578962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-68ae3972-a0fd-43f8-8c7f-891bceecfc2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:39.588348 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98882ec-75d2-469f-b726-978e1b7d1343 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:39.699585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec5eb80-65f1-4af5-8d83-0e75fe671a56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:39.700944 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec5eb80-65f1-4af5-8d83-0e75fe671a56 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:39.701121 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec5eb80-65f1-4af5-8d83-0e75fe671a56 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:39.701520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec5eb80-65f1-4af5-8d83-0e75fe671a56 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:39.703298 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec5eb80-65f1-4af5-8d83-0e75fe671a56 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:39.706430 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec5eb80-65f1-4af5-8d83-0e75fe671a56 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:39.739686 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 669/1354] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:39 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:56:39.764577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ae3972-a0fd-43f8-8c7f-891bceecfc2b None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:39.768674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-68ae3972-a0fd-43f8-8c7f-891bceecfc2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:39.792867 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 686/1355] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:56:39 2022] POST /identity/v3/users => generated 282 bytes in 341 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:39.793817 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec5eb80-65f1-4af5-8d83-0e75fe671a56 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:39.816973 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec5eb80-65f1-4af5-8d83-0e75fe671a56 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:39.830047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5e3a3c-7fa0-41bc-9968-27fc6b567cc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:39.830684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5e3a3c-7fa0-41bc-9968-27fc6b567cc8 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:39.830955 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5e3a3c-7fa0-41bc-9968-27fc6b567cc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:39.831225 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5e3a3c-7fa0-41bc-9968-27fc6b567cc8 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:39.831898 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5e3a3c-7fa0-41bc-9968-27fc6b567cc8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:39.837375 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5e3a3c-7fa0-41bc-9968-27fc6b567cc8 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:39.859257 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 687/1356] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:39 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:39.868620 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5e3a3c-7fa0-41bc-9968-27fc6b567cc8 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:39.882542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:39.891914 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5e3a3c-7fa0-41bc-9968-27fc6b567cc8 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:39.908817 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84633fa2-687a-49c9-9501-53171e226503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:39.909379 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84633fa2-687a-49c9-9501-53171e226503 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:39.909658 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84633fa2-687a-49c9-9501-53171e226503 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:39.909957 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84633fa2-687a-49c9-9501-53171e226503 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:39.910577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84633fa2-687a-49c9-9501-53171e226503 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:39.913425 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84633fa2-687a-49c9-9501-53171e226503 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:39.916373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ae3972-a0fd-43f8-8c7f-891bceecfc2b None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:39.926758 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140f336a-d833-4c4a-88ce-0b1a8b1660e1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:39.927398 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140f336a-d833-4c4a-88ce-0b1a8b1660e1 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:39.927709 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140f336a-d833-4c4a-88ce-0b1a8b1660e1 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:39.927985 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140f336a-d833-4c4a-88ce-0b1a8b1660e1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:39.946585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 688/1357] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:39 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:39.949192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84633fa2-687a-49c9-9501-53171e226503 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:39.955206 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140f336a-d833-4c4a-88ce-0b1a8b1660e1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:39.957525 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140f336a-d833-4c4a-88ce-0b1a8b1660e1 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:39.972070 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84633fa2-687a-49c9-9501-53171e226503 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:40.000412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 670/1358] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:39 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 237 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:40.001226 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b638dd56-efaa-4ad6-b34f-b77fec87a54a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:40.003740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b638dd56-efaa-4ad6-b34f-b77fec87a54a admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:40.004036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b638dd56-efaa-4ad6-b34f-b77fec87a54a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:40.004310 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b638dd56-efaa-4ad6-b34f-b77fec87a54a admin admin] PATH_INFO: `/v3/projects/a95e80610b6f49cb85c7b2f7706056d2/users/7b4a8dfd5e5441eb881b1320d54f9bf4/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:40.074394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b638dd56-efaa-4ad6-b34f-b77fec87a54a admin admin] RBAC: Authorizing `identity:create_grant(project_id=a95e80610b6f49cb85c7b2f7706056d2, user_id=7b4a8dfd5e5441eb881b1320d54f9bf4, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:40.082340 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b638dd56-efaa-4ad6-b34f-b77fec87a54a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:40.130823 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 689/1359] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:39 2022] PUT /identity/v3/projects/a95e80610b6f49cb85c7b2f7706056d2/users/7b4a8dfd5e5441eb881b1320d54f9bf4/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:40.142996 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd2c488-6efd-4830-bce9-52c7f8b90dd5 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:40.143349 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd2c488-6efd-4830-bce9-52c7f8b90dd5 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:40.143544 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd2c488-6efd-4830-bce9-52c7f8b90dd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:40.228711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-cdd2c488-6efd-4830-bce9-52c7f8b90dd5 None None] MFA Rules not processed for user `7b4a8dfd5e5441eb881b1320d54f9bf4`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:40.229545 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b638dd56-efaa-4ad6-b34f-b77fec87a54a admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:40.233793 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-cdd2c488-6efd-4830-bce9-52c7f8b90dd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:40.384746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 671/1360] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:40 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:40.386815 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd2c488-6efd-4830-bce9-52c7f8b90dd5 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:40.402130 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b638dd56-efaa-4ad6-b34f-b77fec87a54a admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:40.414421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798a0d93-9de2-4d5f-b256-58ae4e987779 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:40.414706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798a0d93-9de2-4d5f-b256-58ae4e987779 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:40.415103 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798a0d93-9de2-4d5f-b256-58ae4e987779 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:40.416090 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798a0d93-9de2-4d5f-b256-58ae4e987779 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:40.421375 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-798a0d93-9de2-4d5f-b256-58ae4e987779 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:40.428920 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:40.602278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798a0d93-9de2-4d5f-b256-58ae4e987779 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:40.603965 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798a0d93-9de2-4d5f-b256-58ae4e987779 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:40.637327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 690/1361] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:40 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:56:40.639176 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec81b49d-0b78-4044-993a-d365e839f64c None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:40.650049 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec81b49d-0b78-4044-993a-d365e839f64c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:40.650400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec81b49d-0b78-4044-993a-d365e839f64c None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:40.657921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd2c488-6efd-4830-bce9-52c7f8b90dd5 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:40.659546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 691/1362] 10.222.0.36 () {54 vars in 912 bytes} [Mon Oct 31 14:56:40 2022] GET /identity => generated 270 bytes in 22 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:56:40.661186 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058e85f5-62a7-4d37-bf34-6e75c9e1437e None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:40.661340 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058e85f5-62a7-4d37-bf34-6e75c9e1437e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:40.661513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058e85f5-62a7-4d37-bf34-6e75c9e1437e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:40.667772 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe1a599-4843-4b1c-87bb-515f839091f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:40.670598 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe1a599-4843-4b1c-87bb-515f839091f1 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:40.670986 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe1a599-4843-4b1c-87bb-515f839091f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:40.671354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe1a599-4843-4b1c-87bb-515f839091f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:40.672359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe1a599-4843-4b1c-87bb-515f839091f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:40.677234 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe1a599-4843-4b1c-87bb-515f839091f1 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:40.744272 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 672/1363] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:56:40 2022] POST /identity/v3/projects => generated 363 bytes in 358 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:40.749548 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe1a599-4843-4b1c-87bb-515f839091f1 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:40.781457 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-058e85f5-62a7-4d37-bf34-6e75c9e1437e None None] MFA Rules not processed for user `14946d97fd384e2a8bff68c4f53c2cfe`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:40.788013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-058e85f5-62a7-4d37-bf34-6e75c9e1437e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:40.928749 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe1a599-4843-4b1c-87bb-515f839091f1 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:40.941479 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6151a3c3-1833-49cd-ba26-e21fff6b576c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:40.944109 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6151a3c3-1833-49cd-ba26-e21fff6b576c admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:40.944500 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6151a3c3-1833-49cd-ba26-e21fff6b576c admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:40.946560 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6151a3c3-1833-49cd-ba26-e21fff6b576c admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:40.947450 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6151a3c3-1833-49cd-ba26-e21fff6b576c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:40.952305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6151a3c3-1833-49cd-ba26-e21fff6b576c admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:40.969570 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 692/1364] 10.222.0.36 () {58 vars in 1009 bytes} [Mon Oct 31 14:56:40 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:41.070347 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058e85f5-62a7-4d37-bf34-6e75c9e1437e None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:41.074480 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 673/1365] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:56:40 2022] POST /identity/v3/users => generated 288 bytes in 323 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:56:41.074655 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-058e85f5-62a7-4d37-bf34-6e75c9e1437e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:41.076656 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6151a3c3-1833-49cd-ba26-e21fff6b576c admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:41.094933 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058e85f5-62a7-4d37-bf34-6e75c9e1437e None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:41.107347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6151a3c3-1833-49cd-ba26-e21fff6b576c admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:41.115024 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498a5a1c-99ae-44a2-9284-51cd9e62aed7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:41.115938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498a5a1c-99ae-44a2-9284-51cd9e62aed7 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:41.117650 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498a5a1c-99ae-44a2-9284-51cd9e62aed7 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:41.118119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498a5a1c-99ae-44a2-9284-51cd9e62aed7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:41.124066 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1984b41a-f773-413f-a49f-3ac46ed81dd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:41.124869 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1984b41a-f773-413f-a49f-3ac46ed81dd5 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:41.125241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1984b41a-f773-413f-a49f-3ac46ed81dd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:41.125533 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1984b41a-f773-413f-a49f-3ac46ed81dd5 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:41.126231 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1984b41a-f773-413f-a49f-3ac46ed81dd5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:41.128030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1984b41a-f773-413f-a49f-3ac46ed81dd5 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:41.141712 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498a5a1c-99ae-44a2-9284-51cd9e62aed7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:41.145354 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498a5a1c-99ae-44a2-9284-51cd9e62aed7 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:41.146002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 674/1366] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:41 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:41.148921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1984b41a-f773-413f-a49f-3ac46ed81dd5 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:41.164207 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 693/1367] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:41 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:56:41.166714 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1984b41a-f773-413f-a49f-3ac46ed81dd5 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:41.193728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2fdde9-bdc6-44b2-8d13-f2921482622b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:41.194714 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2fdde9-bdc6-44b2-8d13-f2921482622b admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:41.195148 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2fdde9-bdc6-44b2-8d13-f2921482622b admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:41.195600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2fdde9-bdc6-44b2-8d13-f2921482622b admin admin] PATH_INFO: `/v3/projects/f9de63c41ca84694b34482bf0d56e08d/users/01f207b30ff24fb2b25ae6c4fb7d8dde/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:41.235715 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2fdde9-bdc6-44b2-8d13-f2921482622b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f9de63c41ca84694b34482bf0d56e08d, user_id=01f207b30ff24fb2b25ae6c4fb7d8dde, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:41.238771 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2fdde9-bdc6-44b2-8d13-f2921482622b admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:41.274451 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 675/1368] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:41 2022] PUT /identity/v3/projects/f9de63c41ca84694b34482bf0d56e08d/users/01f207b30ff24fb2b25ae6c4fb7d8dde/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:41.278666 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498a5a1c-99ae-44a2-9284-51cd9e62aed7 service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:41.299140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498a5a1c-99ae-44a2-9284-51cd9e62aed7 service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:41.306041 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:41.454637 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e2408d-81b4-4403-acae-e03e7aa39a4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:41.455420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e2408d-81b4-4403-acae-e03e7aa39a4a admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:41.455729 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e2408d-81b4-4403-acae-e03e7aa39a4a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:41.456003 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e2408d-81b4-4403-acae-e03e7aa39a4a admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:41.456653 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e2408d-81b4-4403-acae-e03e7aa39a4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:41.458316 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e2408d-81b4-4403-acae-e03e7aa39a4a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:41.482469 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 694/1369] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:41 2022] GET /identity/v3/roles => generated 1420 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:41.487662 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2fdde9-bdc6-44b2-8d13-f2921482622b admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:41.501919 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2fdde9-bdc6-44b2-8d13-f2921482622b admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:41.509817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a381a561-5622-43a3-85f8-5e9b083e2de5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:41.510509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a381a561-5622-43a3-85f8-5e9b083e2de5 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:41.510789 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a381a561-5622-43a3-85f8-5e9b083e2de5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:41.512964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a381a561-5622-43a3-85f8-5e9b083e2de5 admin admin] PATH_INFO: `/v3/projects/f9de63c41ca84694b34482bf0d56e08d/users/01f207b30ff24fb2b25ae6c4fb7d8dde/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:41.516504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a381a561-5622-43a3-85f8-5e9b083e2de5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f9de63c41ca84694b34482bf0d56e08d, user_id=01f207b30ff24fb2b25ae6c4fb7d8dde, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:41.522193 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a381a561-5622-43a3-85f8-5e9b083e2de5 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:41.540446 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 676/1370] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:41 2022] PUT /identity/v3/projects/f9de63c41ca84694b34482bf0d56e08d/users/01f207b30ff24fb2b25ae6c4fb7d8dde/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:41.551333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa0fbc0-a7d2-46d9-a931-8d4138c6e1cd None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:41.551668 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa0fbc0-a7d2-46d9-a931-8d4138c6e1cd None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:41.551947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa0fbc0-a7d2-46d9-a931-8d4138c6e1cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:41.617604 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-2aa0fbc0-a7d2-46d9-a931-8d4138c6e1cd None None] MFA Rules not processed for user `01f207b30ff24fb2b25ae6c4fb7d8dde`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:41.618933 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2aa0fbc0-a7d2-46d9-a931-8d4138c6e1cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:41.706684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b9935d-b0d7-4249-97f7-a36f3396c35d None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:41.706684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b9935d-b0d7-4249-97f7-a36f3396c35d None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:41.706684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b9935d-b0d7-4249-97f7-a36f3396c35d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:41.790023 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-d5b9935d-b0d7-4249-97f7-a36f3396c35d None None] MFA Rules not processed for user `e1fb0b1657cd43baaa93a16ce6d9ec8f`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:41.791419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d5b9935d-b0d7-4249-97f7-a36f3396c35d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:41.801576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 695/1371] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:41 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:41.816624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa0fbc0-a7d2-46d9-a931-8d4138c6e1cd None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:41.824254 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2aa0fbc0-a7d2-46d9-a931-8d4138c6e1cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:41.852347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:41.936931 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 677/1372] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:41 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:41.937622 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b9935d-b0d7-4249-97f7-a36f3396c35d None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:41.939100 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d5b9935d-b0d7-4249-97f7-a36f3396c35d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:42.002349 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa0fbc0-a7d2-46d9-a931-8d4138c6e1cd None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:42.011871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89dcc3c-0cdf-43d9-be6d-3a6ed35b5f7a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:42.012691 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89dcc3c-0cdf-43d9-be6d-3a6ed35b5f7a service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:42.013195 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89dcc3c-0cdf-43d9-be6d-3a6ed35b5f7a service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:42.016871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89dcc3c-0cdf-43d9-be6d-3a6ed35b5f7a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:42.035323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89dcc3c-0cdf-43d9-be6d-3a6ed35b5f7a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:42.036978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89dcc3c-0cdf-43d9-be6d-3a6ed35b5f7a service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:42.055779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 696/1373] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:41 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:56:42.056861 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89dcc3c-0cdf-43d9-be6d-3a6ed35b5f7a service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:42.059122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d89dcc3c-0cdf-43d9-be6d-3a6ed35b5f7a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:42.067588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b9935d-b0d7-4249-97f7-a36f3396c35d None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:42.081707 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2222b9-f0b4-434e-830e-d0cc824c90f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:42.082626 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2222b9-f0b4-434e-830e-d0cc824c90f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:42.082913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2222b9-f0b4-434e-830e-d0cc824c90f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:42.083188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2222b9-f0b4-434e-830e-d0cc824c90f5 admin admin] PATH_INFO: `/v3/users/94d17cbf9c7d49da8cd289c0d1c390e8` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:42.084747 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2222b9-f0b4-434e-830e-d0cc824c90f5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=94d17cbf9c7d49da8cd289c0d1c390e8)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:42.088210 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2222b9-f0b4-434e-830e-d0cc824c90f5 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:42.173801 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:42.202610 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5f2222b9-f0b4-434e-830e-d0cc824c90f5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '94d17cbf9c7d49da8cd289c0d1c390e8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:42.279002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5f2222b9-f0b4-434e-830e-d0cc824c90f5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '94d17cbf9c7d49da8cd289c0d1c390e8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:42.280856 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5f2222b9-f0b4-434e-830e-d0cc824c90f5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '94d17cbf9c7d49da8cd289c0d1c390e8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:42.281585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5f2222b9-f0b4-434e-830e-d0cc824c90f5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '94d17cbf9c7d49da8cd289c0d1c390e8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:42.324915 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5f2222b9-f0b4-434e-830e-d0cc824c90f5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '94d17cbf9c7d49da8cd289c0d1c390e8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:42.348747 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89dcc3c-0cdf-43d9-be6d-3a6ed35b5f7a service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:42.362151 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 678/1374] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:56:41 2022] DELETE /identity/v3/users/94d17cbf9c7d49da8cd289c0d1c390e8 => generated 0 bytes in 425 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:42.362765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2222b9-f0b4-434e-830e-d0cc824c90f5 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:42.365420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5f2222b9-f0b4-434e-830e-d0cc824c90f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:42.376343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b02d8b4-c15d-4a9c-a574-a3a41c021477 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:42.377193 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b02d8b4-c15d-4a9c-a574-a3a41c021477 service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:42.377631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b02d8b4-c15d-4a9c-a574-a3a41c021477 service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:42.378063 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b02d8b4-c15d-4a9c-a574-a3a41c021477 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:42.389744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7b02d8b4-c15d-4a9c-a574-a3a41c021477 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:42.408481 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b02d8b4-c15d-4a9c-a574-a3a41c021477 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:42.412214 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b02d8b4-c15d-4a9c-a574-a3a41c021477 service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:42.426280 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 697/1375] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:42 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:42.428367 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e632131-c027-4eb3-9e2e-0eb809b58070 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:42.428958 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e632131-c027-4eb3-9e2e-0eb809b58070 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:42.429441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e632131-c027-4eb3-9e2e-0eb809b58070 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:42.475320 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:42.480824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-2e632131-c027-4eb3-9e2e-0eb809b58070 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:42.483123 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2e632131-c027-4eb3-9e2e-0eb809b58070 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:42.514146 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2222b9-f0b4-434e-830e-d0cc824c90f5 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:42.518062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacbcf3a-3969-4e11-b0e5-c0cd19b1c9d2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:42.519151 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacbcf3a-3969-4e11-b0e5-c0cd19b1c9d2 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:42.519436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacbcf3a-3969-4e11-b0e5-c0cd19b1c9d2 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:42.519750 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacbcf3a-3969-4e11-b0e5-c0cd19b1c9d2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:42.530147 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-aacbcf3a-3969-4e11-b0e5-c0cd19b1c9d2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:42.677937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 698/1376] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:42.685734 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e632131-c027-4eb3-9e2e-0eb809b58070 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:42.687547 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2e632131-c027-4eb3-9e2e-0eb809b58070 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:42.697776 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacbcf3a-3969-4e11-b0e5-c0cd19b1c9d2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:42.702158 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacbcf3a-3969-4e11-b0e5-c0cd19b1c9d2 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:42.733308 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 679/1377] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:42 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:42.802109 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdd6666-c0e4-44fc-bc53-d73a55378b0c None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:42.802476 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdd6666-c0e4-44fc-bc53-d73a55378b0c None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:42.803369 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdd6666-c0e4-44fc-bc53-d73a55378b0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:42.845099 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e632131-c027-4eb3-9e2e-0eb809b58070 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:42.860415 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6ad345-7de7-4f7f-93ba-2da648b0926d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:42.861447 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6ad345-7de7-4f7f-93ba-2da648b0926d service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:42.861447 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6ad345-7de7-4f7f-93ba-2da648b0926d service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:42.861911 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6ad345-7de7-4f7f-93ba-2da648b0926d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:42.872490 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-4fdd6666-c0e4-44fc-bc53-d73a55378b0c None None] MFA Rules not processed for user `75d1901be8064ff5bf79f0268d589e10`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:42.873866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-4fdd6666-c0e4-44fc-bc53-d73a55378b0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:42.891446 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6ad345-7de7-4f7f-93ba-2da648b0926d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:42.896404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6ad345-7de7-4f7f-93ba-2da648b0926d service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:42.929500 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 699/1378] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:56:43.114882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 680/1379] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:42 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 314 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:43.130796 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6ad345-7de7-4f7f-93ba-2da648b0926d service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:43.132550 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0b6ad345-7de7-4f7f-93ba-2da648b0926d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:43.384995 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:43.448684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6ad345-7de7-4f7f-93ba-2da648b0926d service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:43.473114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4874a42-936c-4a1c-a038-c014144087a0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:43.473739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4874a42-936c-4a1c-a038-c014144087a0 service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:43.477538 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4874a42-936c-4a1c-a038-c014144087a0 service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:43.477845 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4874a42-936c-4a1c-a038-c014144087a0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:43.513404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4874a42-936c-4a1c-a038-c014144087a0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:43.517744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4874a42-936c-4a1c-a038-c014144087a0 service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:43.563928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 700/1380] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:43 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:43.953507 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdd6666-c0e4-44fc-bc53-d73a55378b0c None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:43.955010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-4fdd6666-c0e4-44fc-bc53-d73a55378b0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:44.150651 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdd6666-c0e4-44fc-bc53-d73a55378b0c None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:44.166286 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc12534b-d15f-4eb4-a2cc-c2e18fddd8db service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70ea2aaf529b46dabb70b71f4dcc3860', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:44.168297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc12534b-d15f-4eb4-a2cc-c2e18fddd8db service placement] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:44.169643 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc12534b-d15f-4eb4-a2cc-c2e18fddd8db service placement] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:44.170138 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc12534b-d15f-4eb4-a2cc-c2e18fddd8db service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:44.173639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-cc12534b-d15f-4eb4-a2cc-c2e18fddd8db service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:44.211797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc12534b-d15f-4eb4-a2cc-c2e18fddd8db service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:44.215352 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc12534b-d15f-4eb4-a2cc-c2e18fddd8db service placement] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:44.271063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 681/1381] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:43 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:44.447012 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4874a42-936c-4a1c-a038-c014144087a0 service nova] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:44.449632 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-f4874a42-936c-4a1c-a038-c014144087a0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:44.475402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4874a42-936c-4a1c-a038-c014144087a0 service nova] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:44.490646 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8812869e-4736-4b06-a619-5d832ac1561b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:44.491749 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8812869e-4736-4b06-a619-5d832ac1561b admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:44.492231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8812869e-4736-4b06-a619-5d832ac1561b admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:44.492706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8812869e-4736-4b06-a619-5d832ac1561b admin admin] PATH_INFO: `/v3/projects/009f7c73f7dc486880989c38883379dd` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:44.494166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8812869e-4736-4b06-a619-5d832ac1561b admin admin] RBAC: Authorizing `identity:delete_project(project_id=009f7c73f7dc486880989c38883379dd)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:44.498537 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8812869e-4736-4b06-a619-5d832ac1561b admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:44.640685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8812869e-4736-4b06-a619-5d832ac1561b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '009f7c73f7dc486880989c38883379dd', 'request_id': 'req-8812869e-4736-4b06-a619-5d832ac1561b'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:44.653127 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8812869e-4736-4b06-a619-5d832ac1561b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '009f7c73f7dc486880989c38883379dd', 'request_id': 'req-8812869e-4736-4b06-a619-5d832ac1561b'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:44.672080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8812869e-4736-4b06-a619-5d832ac1561b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '009f7c73f7dc486880989c38883379dd', 'request_id': 'req-8812869e-4736-4b06-a619-5d832ac1561b'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:44.697135 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8812869e-4736-4b06-a619-5d832ac1561b admin admin] The token cache is being invalidate because project 009f7c73f7dc486880989c38883379dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:56:44.701861 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8812869e-4736-4b06-a619-5d832ac1561b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:44.704869 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8812869e-4736-4b06-a619-5d832ac1561b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:44.712457 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 701/1382] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:56:44 2022] DELETE /identity/v3/projects/009f7c73f7dc486880989c38883379dd => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:44.717088 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc12534b-d15f-4eb4-a2cc-c2e18fddd8db service placement] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:44.734743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc12534b-d15f-4eb4-a2cc-c2e18fddd8db service placement] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:44.775397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:44.890099 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ac8fc7-160a-4444-a3b3-aebe412c7960 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:44.890934 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ac8fc7-160a-4444-a3b3-aebe412c7960 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:44.891770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ac8fc7-160a-4444-a3b3-aebe412c7960 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:44.892191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ac8fc7-160a-4444-a3b3-aebe412c7960 admin admin] PATH_INFO: `/v3/users/10f72295f40a4af9ab2219cc59b63ab6` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:44.894508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ac8fc7-160a-4444-a3b3-aebe412c7960 admin admin] RBAC: Authorizing `identity:delete_user(user_id=10f72295f40a4af9ab2219cc59b63ab6)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:44.898317 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ac8fc7-160a-4444-a3b3-aebe412c7960 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:45.052059 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-80ac8fc7-160a-4444-a3b3-aebe412c7960 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '10f72295f40a4af9ab2219cc59b63ab6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:45.072565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-80ac8fc7-160a-4444-a3b3-aebe412c7960 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '10f72295f40a4af9ab2219cc59b63ab6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:45.073487 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-80ac8fc7-160a-4444-a3b3-aebe412c7960 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '10f72295f40a4af9ab2219cc59b63ab6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:45.079001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-80ac8fc7-160a-4444-a3b3-aebe412c7960 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '10f72295f40a4af9ab2219cc59b63ab6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:45.128767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-80ac8fc7-160a-4444-a3b3-aebe412c7960 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '10f72295f40a4af9ab2219cc59b63ab6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:45.159813 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 682/1383] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:56:44 2022] DELETE /identity/v3/users/10f72295f40a4af9ab2219cc59b63ab6 => generated 0 bytes in 443 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:56:47.075048 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8812869e-4736-4b06-a619-5d832ac1561b admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:47.077825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8812869e-4736-4b06-a619-5d832ac1561b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:47.093951 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8812869e-4736-4b06-a619-5d832ac1561b admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:47.222682 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bdca37-829e-467f-a381-6e39eb4a5374 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:47.223585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bdca37-829e-467f-a381-6e39eb4a5374 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:47.224007 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bdca37-829e-467f-a381-6e39eb4a5374 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:47.224571 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bdca37-829e-467f-a381-6e39eb4a5374 admin admin] PATH_INFO: `/v3/projects/5c2b1a5357ae42b9b12418dd405816c7` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:47.226934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bdca37-829e-467f-a381-6e39eb4a5374 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c2b1a5357ae42b9b12418dd405816c7)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:47.229414 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bdca37-829e-467f-a381-6e39eb4a5374 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:47.386863 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f6bdca37-829e-467f-a381-6e39eb4a5374 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c2b1a5357ae42b9b12418dd405816c7', 'request_id': 'req-f6bdca37-829e-467f-a381-6e39eb4a5374'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:47.405958 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f6bdca37-829e-467f-a381-6e39eb4a5374 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c2b1a5357ae42b9b12418dd405816c7', 'request_id': 'req-f6bdca37-829e-467f-a381-6e39eb4a5374'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:47.435662 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f6bdca37-829e-467f-a381-6e39eb4a5374 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c2b1a5357ae42b9b12418dd405816c7', 'request_id': 'req-f6bdca37-829e-467f-a381-6e39eb4a5374'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:47.458903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:47.469283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f6bdca37-829e-467f-a381-6e39eb4a5374 admin admin] The token cache is being invalidate because project 5c2b1a5357ae42b9b12418dd405816c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:56:47.473419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f6bdca37-829e-467f-a381-6e39eb4a5374 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:47.476654 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f6bdca37-829e-467f-a381-6e39eb4a5374 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:47.486311 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 702/1384] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:56:47 2022] DELETE /identity/v3/projects/5c2b1a5357ae42b9b12418dd405816c7 => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:47.487932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9fa3cb-e391-4f75-a7a5-2de48606b3ef None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:47.488730 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9fa3cb-e391-4f75-a7a5-2de48606b3ef None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:47.489167 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9fa3cb-e391-4f75-a7a5-2de48606b3ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:47.537064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-5d9fa3cb-e391-4f75-a7a5-2de48606b3ef None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:47.538523 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5d9fa3cb-e391-4f75-a7a5-2de48606b3ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:47.735920 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 683/1385] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 249 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:56:47.743773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c79ca2f-3164-4144-8d92-20f967a00599 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:47.744063 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c79ca2f-3164-4144-8d92-20f967a00599 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:47.744341 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c79ca2f-3164-4144-8d92-20f967a00599 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:47.834993 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-3c79ca2f-3164-4144-8d92-20f967a00599 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:47.836283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-3c79ca2f-3164-4144-8d92-20f967a00599 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:47.870574 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 703/1386] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:47.871690 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9fa3cb-e391-4f75-a7a5-2de48606b3ef None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:47.889066 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9fa3cb-e391-4f75-a7a5-2de48606b3ef None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:47.910771 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ca7ccc-17ee-4a92-9de2-34b8663f600d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:47.912600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ca7ccc-17ee-4a92-9de2-34b8663f600d admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:47.912910 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ca7ccc-17ee-4a92-9de2-34b8663f600d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:47.913689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ca7ccc-17ee-4a92-9de2-34b8663f600d admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:47.915806 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ca7ccc-17ee-4a92-9de2-34b8663f600d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:56:47.916171 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ca7ccc-17ee-4a92-9de2-34b8663f600d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:47.920935 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ca7ccc-17ee-4a92-9de2-34b8663f600d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:47.971307 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 684/1387] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:56:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:56:47.975530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c79ca2f-3164-4144-8d92-20f967a00599 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:47.988463 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c79ca2f-3164-4144-8d92-20f967a00599 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:48.002125 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09a40b2-4890-46ba-8004-f35752225a55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:48.002773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09a40b2-4890-46ba-8004-f35752225a55 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:48.003062 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09a40b2-4890-46ba-8004-f35752225a55 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:48.003339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09a40b2-4890-46ba-8004-f35752225a55 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:48.004032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09a40b2-4890-46ba-8004-f35752225a55 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:48.006755 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09a40b2-4890-46ba-8004-f35752225a55 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:48.078502 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 704/1388] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:56:47 2022] POST /identity/v3/projects => generated 381 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:56:48.083858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ca7ccc-17ee-4a92-9de2-34b8663f600d admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:48.113730 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ca7ccc-17ee-4a92-9de2-34b8663f600d admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:48.234803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:48.388814 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1caf384-6a8f-4eeb-925b-39bcd5421316 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:48.389867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1caf384-6a8f-4eeb-925b-39bcd5421316 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:48.390848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1caf384-6a8f-4eeb-925b-39bcd5421316 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:48.390848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1caf384-6a8f-4eeb-925b-39bcd5421316 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:48.391432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1caf384-6a8f-4eeb-925b-39bcd5421316 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:48.394269 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1caf384-6a8f-4eeb-925b-39bcd5421316 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:48.438017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356a01e5-8f46-44b9-af7e-4d069ccf3a07 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:48.438017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356a01e5-8f46-44b9-af7e-4d069ccf3a07 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:48.438017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356a01e5-8f46-44b9-af7e-4d069ccf3a07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:48.478094 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 685/1389] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:56:48 2022] POST /identity/v3/users => generated 291 bytes in 395 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:48.483809 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1caf384-6a8f-4eeb-925b-39bcd5421316 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:48.504058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-356a01e5-8f46-44b9-af7e-4d069ccf3a07 None None] MFA Rules not processed for user `e1fb0b1657cd43baaa93a16ce6d9ec8f`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:48.506070 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-356a01e5-8f46-44b9-af7e-4d069ccf3a07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:48.508989 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1caf384-6a8f-4eeb-925b-39bcd5421316 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:48.531586 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab67e9a-534f-4e51-bd16-050863bd9b24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:48.532658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab67e9a-534f-4e51-bd16-050863bd9b24 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:48.533157 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab67e9a-534f-4e51-bd16-050863bd9b24 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:48.533620 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab67e9a-534f-4e51-bd16-050863bd9b24 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:48.549347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab67e9a-534f-4e51-bd16-050863bd9b24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:48.555992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab67e9a-534f-4e51-bd16-050863bd9b24 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:48.570668 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 686/1390] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:48 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:48.582608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab67e9a-534f-4e51-bd16-050863bd9b24 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:48.634360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab67e9a-534f-4e51-bd16-050863bd9b24 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:48.641518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4f4009-1517-40d2-a3c4-7cf6ac5c1b4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:48.642653 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4f4009-1517-40d2-a3c4-7cf6ac5c1b4d admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:48.643116 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4f4009-1517-40d2-a3c4-7cf6ac5c1b4d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:48.643633 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4f4009-1517-40d2-a3c4-7cf6ac5c1b4d admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:48.644692 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4f4009-1517-40d2-a3c4-7cf6ac5c1b4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:48.647438 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4f4009-1517-40d2-a3c4-7cf6ac5c1b4d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:48.665582 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 687/1391] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:48 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:48.666774 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4f4009-1517-40d2-a3c4-7cf6ac5c1b4d admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:48.688765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4f4009-1517-40d2-a3c4-7cf6ac5c1b4d admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:48.694649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a716ea-1f18-43b4-b9f0-e8e134aaa6a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:48.695711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a716ea-1f18-43b4-b9f0-e8e134aaa6a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:48.707828 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a716ea-1f18-43b4-b9f0-e8e134aaa6a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:48.710696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a716ea-1f18-43b4-b9f0-e8e134aaa6a6 admin admin] PATH_INFO: `/v3/projects/32b4f28185d74f9794ce9d7f1feb4445/users/04456746d23c484a8196387440affe3e/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:48.757368 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 705/1392] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:48 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 321 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:48.792607 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a716ea-1f18-43b4-b9f0-e8e134aaa6a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=32b4f28185d74f9794ce9d7f1feb4445, user_id=04456746d23c484a8196387440affe3e, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:48.803414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a716ea-1f18-43b4-b9f0-e8e134aaa6a6 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:48.854223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 688/1393] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:48 2022] PUT /identity/v3/projects/32b4f28185d74f9794ce9d7f1feb4445/users/04456746d23c484a8196387440affe3e/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:48.865738 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9b9e2e-a8d3-4995-ab67-203a7b1e0408 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:48.866067 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9b9e2e-a8d3-4995-ab67-203a7b1e0408 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:48.866344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9b9e2e-a8d3-4995-ab67-203a7b1e0408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:48.948952 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-af9b9e2e-a8d3-4995-ab67-203a7b1e0408 None None] MFA Rules not processed for user `04456746d23c484a8196387440affe3e`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:48.954476 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-af9b9e2e-a8d3-4995-ab67-203a7b1e0408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:49.160458 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 706/1394] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:48 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 296 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:49.162911 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a716ea-1f18-43b4-b9f0-e8e134aaa6a6 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:49.424222 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a716ea-1f18-43b4-b9f0-e8e134aaa6a6 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:49.439429 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bbb72c-7433-4d62-bd2c-abcc316bc643 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:49.440133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bbb72c-7433-4d62-bd2c-abcc316bc643 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:49.440424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bbb72c-7433-4d62-bd2c-abcc316bc643 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:49.440703 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bbb72c-7433-4d62-bd2c-abcc316bc643 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:49.441366 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bbb72c-7433-4d62-bd2c-abcc316bc643 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:49.445421 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bbb72c-7433-4d62-bd2c-abcc316bc643 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:49.540402 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 689/1395] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:56:49 2022] POST /identity/v3/projects => generated 379 bytes in 375 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:49.543330 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9b9e2e-a8d3-4995-ab67-203a7b1e0408 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:49.571403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9b9e2e-a8d3-4995-ab67-203a7b1e0408 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:49.779497 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f521260-7c23-4b03-80d9-e93429ba0e91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:49.781470 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f521260-7c23-4b03-80d9-e93429ba0e91 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:49.781766 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f521260-7c23-4b03-80d9-e93429ba0e91 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:49.782045 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f521260-7c23-4b03-80d9-e93429ba0e91 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:49.783283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f521260-7c23-4b03-80d9-e93429ba0e91 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:49.787822 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f521260-7c23-4b03-80d9-e93429ba0e91 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:49.802936 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:49.833663 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 707/1396] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:56:49 2022] POST /identity/v3/users => generated 296 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:49.835343 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bbb72c-7433-4d62-bd2c-abcc316bc643 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:49.869322 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bbb72c-7433-4d62-bd2c-abcc316bc643 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:49.874494 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbe4e27-2c6a-47df-818f-fc8a3095837d None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:49.874786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbe4e27-2c6a-47df-818f-fc8a3095837d None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:49.875065 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbe4e27-2c6a-47df-818f-fc8a3095837d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:49.893147 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fb22d0-8d61-40df-8df3-18ad4f06208c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:49.894175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fb22d0-8d61-40df-8df3-18ad4f06208c admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:49.894822 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fb22d0-8d61-40df-8df3-18ad4f06208c admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:49.895236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fb22d0-8d61-40df-8df3-18ad4f06208c admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:49.898241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fb22d0-8d61-40df-8df3-18ad4f06208c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:49.905863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fb22d0-8d61-40df-8df3-18ad4f06208c admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:49.954462 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 690/1397] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:49 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:49.956120 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fb22d0-8d61-40df-8df3-18ad4f06208c admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:49.990581 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-3dbe4e27-2c6a-47df-818f-fc8a3095837d None None] MFA Rules not processed for user `e0d0f8598fac40449bcb2a55ff8df8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:49.994360 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-3dbe4e27-2c6a-47df-818f-fc8a3095837d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:50.000389 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fb22d0-8d61-40df-8df3-18ad4f06208c admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:50.023022 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb967ed-d397-4097-b367-5805cad624a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:50.024053 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb967ed-d397-4097-b367-5805cad624a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:50.028018 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb967ed-d397-4097-b367-5805cad624a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:50.031740 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb967ed-d397-4097-b367-5805cad624a9 admin admin] PATH_INFO: `/v3/projects/acd1ab2e91a74fd089e1a6c539e7961a/users/cffe160549c840ef9b119eaf99111dba/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:50.097664 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb967ed-d397-4097-b367-5805cad624a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=acd1ab2e91a74fd089e1a6c539e7961a, user_id=cffe160549c840ef9b119eaf99111dba, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:50.103361 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb967ed-d397-4097-b367-5805cad624a9 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:50.148573 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 708/1398] 10.222.0.36 () {58 vars in 1009 bytes} [Mon Oct 31 14:56:49 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:56:50.151735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 691/1399] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:49 2022] PUT /identity/v3/projects/acd1ab2e91a74fd089e1a6c539e7961a/users/cffe160549c840ef9b119eaf99111dba/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:50.170417 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbe4e27-2c6a-47df-818f-fc8a3095837d None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:50.201814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbe4e27-2c6a-47df-818f-fc8a3095837d None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:50.331319 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e51830f-87e6-4631-aa5f-513eea56b652 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:50.331995 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e51830f-87e6-4631-aa5f-513eea56b652 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:50.332276 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e51830f-87e6-4631-aa5f-513eea56b652 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:50.332551 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e51830f-87e6-4631-aa5f-513eea56b652 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:50.333201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e51830f-87e6-4631-aa5f-513eea56b652 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:50.335078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e51830f-87e6-4631-aa5f-513eea56b652 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:50.369859 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 709/1400] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:56:50 2022] GET /identity/v3/roles => generated 1420 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:56:50.375013 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb967ed-d397-4097-b367-5805cad624a9 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:50.398069 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb967ed-d397-4097-b367-5805cad624a9 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:50.405519 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb1c983-8453-47f3-9f46-d3313354b703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:50.406645 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb1c983-8453-47f3-9f46-d3313354b703 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:50.408884 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb1c983-8453-47f3-9f46-d3313354b703 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:50.409257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb1c983-8453-47f3-9f46-d3313354b703 admin admin] PATH_INFO: `/v3/projects/acd1ab2e91a74fd089e1a6c539e7961a/users/cffe160549c840ef9b119eaf99111dba/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:50.416145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb1c983-8453-47f3-9f46-d3313354b703 admin admin] RBAC: Authorizing `identity:create_grant(project_id=acd1ab2e91a74fd089e1a6c539e7961a, user_id=cffe160549c840ef9b119eaf99111dba, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:50.419898 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb1c983-8453-47f3-9f46-d3313354b703 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:50.445400 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 692/1401] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:56:50 2022] PUT /identity/v3/projects/acd1ab2e91a74fd089e1a6c539e7961a/users/cffe160549c840ef9b119eaf99111dba/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:50.446951 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83dc6d2-6572-4e58-afaf-359acdb56866 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:50.447419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83dc6d2-6572-4e58-afaf-359acdb56866 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:50.447882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83dc6d2-6572-4e58-afaf-359acdb56866 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:50.463733 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb1c983-8453-47f3-9f46-d3313354b703 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:50.466371 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-cbb1c983-8453-47f3-9f46-d3313354b703 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:50.472649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:50.508814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-a83dc6d2-6572-4e58-afaf-359acdb56866 None None] MFA Rules not processed for user `cffe160549c840ef9b119eaf99111dba`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:50.510325 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a83dc6d2-6572-4e58-afaf-359acdb56866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:50.662586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:50.737093 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 710/1402] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:50 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 292 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:56:50.738820 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786c7588-82e5-4e9f-811b-a21401d85e04 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:50.740968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786c7588-82e5-4e9f-811b-a21401d85e04 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:50.742124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786c7588-82e5-4e9f-811b-a21401d85e04 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:50.743839 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb1c983-8453-47f3-9f46-d3313354b703 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:50.750171 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 711/1403] 10.222.0.36 () {54 vars in 912 bytes} [Mon Oct 31 14:56:50 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:56:50.757082 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfbb570-3d45-4026-b2e4-933a4d24ee17 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:50.757300 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfbb570-3d45-4026-b2e4-933a4d24ee17 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:50.759659 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfbb570-3d45-4026-b2e4-933a4d24ee17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:50.761190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde9320e-e703-4583-b2d6-8c0b185b6cf4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4ae5e20cc5b40ccbdb951f908f5080d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:50.762039 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde9320e-e703-4583-b2d6-8c0b185b6cf4 service swift] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:50.762489 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde9320e-e703-4583-b2d6-8c0b185b6cf4 service swift] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:50.763057 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde9320e-e703-4583-b2d6-8c0b185b6cf4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:50.804695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-bdfbb570-3d45-4026-b2e4-933a4d24ee17 None None] MFA Rules not processed for user `14946d97fd384e2a8bff68c4f53c2cfe`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:50.806193 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-bdfbb570-3d45-4026-b2e4-933a4d24ee17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:50.926917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 712/1404] 10.222.0.36 () {58 vars in 1009 bytes} [Mon Oct 31 14:56:50 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:50.927990 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfbb570-3d45-4026-b2e4-933a4d24ee17 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:50.929834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-bdfbb570-3d45-4026-b2e4-933a4d24ee17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:50.978777 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde9320e-e703-4583-b2d6-8c0b185b6cf4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:50.985123 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde9320e-e703-4583-b2d6-8c0b185b6cf4 service swift] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:51.011697 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 693/1405] 10.222.0.36 () {60 vars in 1384 bytes} [Mon Oct 31 14:56:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 548 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 31 14:56:51.013386 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde9320e-e703-4583-b2d6-8c0b185b6cf4 service swift] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:51.016073 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfbb570-3d45-4026-b2e4-933a4d24ee17 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:51.018599 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-bde9320e-e703-4583-b2d6-8c0b185b6cf4 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:51.028361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf136461-3420-4071-9fd4-b6c2f9957aa6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:51.029464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf136461-3420-4071-9fd4-b6c2f9957aa6 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:51.029963 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf136461-3420-4071-9fd4-b6c2f9957aa6 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:51.030403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf136461-3420-4071-9fd4-b6c2f9957aa6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:51.045821 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf136461-3420-4071-9fd4-b6c2f9957aa6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:51.047444 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf136461-3420-4071-9fd4-b6c2f9957aa6 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:51.047754 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde9320e-e703-4583-b2d6-8c0b185b6cf4 service swift] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:51.068565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f944baf-f97f-4736-8d33-8ac6a3af96a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:51.069964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f944baf-f97f-4736-8d33-8ac6a3af96a3 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:51.071036 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f944baf-f97f-4736-8d33-8ac6a3af96a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:51.072084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f944baf-f97f-4736-8d33-8ac6a3af96a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:51.091627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f944baf-f97f-4736-8d33-8ac6a3af96a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:51.095673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 713/1406] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:50 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:51.107888 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f944baf-f97f-4736-8d33-8ac6a3af96a3 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:51.141998 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 694/1407] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:51 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:51.959410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf136461-3420-4071-9fd4-b6c2f9957aa6 service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:51.975343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf136461-3420-4071-9fd4-b6c2f9957aa6 service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:52.013246 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af392182-14cc-44ec-95b5-1e89e14116ae tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cffe160549c840ef9b119eaf99111dba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'acd1ab2e91a74fd089e1a6c539e7961a', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:52.014917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af392182-14cc-44ec-95b5-1e89e14116ae tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:52.015844 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af392182-14cc-44ec-95b5-1e89e14116ae tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:52.016229 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af392182-14cc-44ec-95b5-1e89e14116ae tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:52.018567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af392182-14cc-44ec-95b5-1e89e14116ae tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:56:52.019311 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af392182-14cc-44ec-95b5-1e89e14116ae tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:52.027891 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af392182-14cc-44ec-95b5-1e89e14116ae tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:52.081824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f944baf-f97f-4736-8d33-8ac6a3af96a3 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:52.083183 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 714/1408] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:56:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:56:52.106418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f944baf-f97f-4736-8d33-8ac6a3af96a3 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:52.119586 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfb4a72-ed13-48cb-8e3a-85d4b5c94385 tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cffe160549c840ef9b119eaf99111dba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'acd1ab2e91a74fd089e1a6c539e7961a', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:52.120305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfb4a72-ed13-48cb-8e3a-85d4b5c94385 tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:52.120679 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfb4a72-ed13-48cb-8e3a-85d4b5c94385 tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:52.120950 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfb4a72-ed13-48cb-8e3a-85d4b5c94385 tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:52.121661 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfb4a72-ed13-48cb-8e3a-85d4b5c94385 tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:52.123873 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfb4a72-ed13-48cb-8e3a-85d4b5c94385 tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:52.193420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 695/1409] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:56:52 2022] POST /identity/v3/projects => generated 355 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:56:52.205505 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af392182-14cc-44ec-95b5-1e89e14116ae tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:52.208888 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fb9bc1-85c3-4bb2-a7fb-02eab482a51d None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:52.209653 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fb9bc1-85c3-4bb2-a7fb-02eab482a51d None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:52.210143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fb9bc1-85c3-4bb2-a7fb-02eab482a51d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:52.231604 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af392182-14cc-44ec-95b5-1e89e14116ae tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:52.277205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-a6fb9bc1-85c3-4bb2-a7fb-02eab482a51d None None] MFA Rules not processed for user `cdcbcb2ed9e44dbe8444e531fc1eb707`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:52.278863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a6fb9bc1-85c3-4bb2-a7fb-02eab482a51d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:52.379144 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:52.441346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 696/1410] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:52 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:52.442852 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dcca33-12b8-4424-97d9-9f770527b369 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:52.443231 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dcca33-12b8-4424-97d9-9f770527b369 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:52.443925 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dcca33-12b8-4424-97d9-9f770527b369 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:52.507616 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-a4dcca33-12b8-4424-97d9-9f770527b369 None None] MFA Rules not processed for user `e6bfe2c1ba054935ae8e9fcc054faf68`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:52.510443 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a4dcca33-12b8-4424-97d9-9f770527b369 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:52.584404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b560459c-a262-4d34-9c3e-5730e306e213 tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cffe160549c840ef9b119eaf99111dba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'acd1ab2e91a74fd089e1a6c539e7961a', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:52.586556 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b560459c-a262-4d34-9c3e-5730e306e213 tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:52.586854 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b560459c-a262-4d34-9c3e-5730e306e213 tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:52.587138 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b560459c-a262-4d34-9c3e-5730e306e213 tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:52.587883 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b560459c-a262-4d34-9c3e-5730e306e213 tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:56:52.588201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b560459c-a262-4d34-9c3e-5730e306e213 tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:52.606091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b560459c-a262-4d34-9c3e-5730e306e213 tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:52.643861 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 715/1411] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:56:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 437 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:56:52.643861 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b560459c-a262-4d34-9c3e-5730e306e213 tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:52.692934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b560459c-a262-4d34-9c3e-5730e306e213 tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:52.724717 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 697/1412] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:52 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:52.738859 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f6c7cd-7d77-4dab-802b-a7ed25baa0e6 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:52.739899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f6c7cd-7d77-4dab-802b-a7ed25baa0e6 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:52.740453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f6c7cd-7d77-4dab-802b-a7ed25baa0e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:52.769658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-d1f6c7cd-7d77-4dab-802b-a7ed25baa0e6 None None] MFA Rules not processed for user `e6bfe2c1ba054935ae8e9fcc054faf68`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:52.773170 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d1f6c7cd-7d77-4dab-802b-a7ed25baa0e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:52.785692 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 698/1413] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:52 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:52.786946 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f6c7cd-7d77-4dab-802b-a7ed25baa0e6 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:52.815509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f6c7cd-7d77-4dab-802b-a7ed25baa0e6 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:52.830244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ab892c-8947-409b-80c0-c003dcd870aa tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cffe160549c840ef9b119eaf99111dba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'acd1ab2e91a74fd089e1a6c539e7961a', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:52.832219 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ab892c-8947-409b-80c0-c003dcd870aa tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:52.832935 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ab892c-8947-409b-80c0-c003dcd870aa tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:52.833411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ab892c-8947-409b-80c0-c003dcd870aa tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] PATH_INFO: `/v3/projects/dfab8ebbcfb641b98d884ba52bbe6380` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:52.835275 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ab892c-8947-409b-80c0-c003dcd870aa tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] RBAC: Authorizing `identity:delete_project(project_id=dfab8ebbcfb641b98d884ba52bbe6380)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:52.839074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ab892c-8947-409b-80c0-c003dcd870aa tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:52.853107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:53.038655 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-04ab892c-8947-409b-80c0-c003dcd870aa tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dfab8ebbcfb641b98d884ba52bbe6380', 'request_id': 'req-04ab892c-8947-409b-80c0-c003dcd870aa'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:53.048112 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-04ab892c-8947-409b-80c0-c003dcd870aa tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dfab8ebbcfb641b98d884ba52bbe6380', 'request_id': 'req-04ab892c-8947-409b-80c0-c003dcd870aa'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:53.057017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-04ab892c-8947-409b-80c0-c003dcd870aa tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dfab8ebbcfb641b98d884ba52bbe6380', 'request_id': 'req-04ab892c-8947-409b-80c0-c003dcd870aa'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:53.096444 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b560459c-a262-4d34-9c3e-5730e306e213 tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:53.104089 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-04ab892c-8947-409b-80c0-c003dcd870aa tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] The token cache is being invalidate because project dfab8ebbcfb641b98d884ba52bbe6380 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:56:53.105236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-04ab892c-8947-409b-80c0-c003dcd870aa tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:53.112361 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-04ab892c-8947-409b-80c0-c003dcd870aa tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:53.133972 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 699/1414] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:56:52 2022] DELETE /identity/v3/projects/dfab8ebbcfb641b98d884ba52bbe6380 => generated 0 bytes in 347 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:53.310116 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8d1304-b01c-48d9-9710-4b64eeaeb8c8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:53.310924 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8d1304-b01c-48d9-9710-4b64eeaeb8c8 service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:53.311521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8d1304-b01c-48d9-9710-4b64eeaeb8c8 service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:53.311918 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8d1304-b01c-48d9-9710-4b64eeaeb8c8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:53.317695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7d8d1304-b01c-48d9-9710-4b64eeaeb8c8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:53.542282 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8d1304-b01c-48d9-9710-4b64eeaeb8c8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:53.545185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8d1304-b01c-48d9-9710-4b64eeaeb8c8 service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:53.579007 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 716/1415] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:52 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 937 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:53.720533 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ab892c-8947-409b-80c0-c003dcd870aa tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:53.724473 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-04ab892c-8947-409b-80c0-c003dcd870aa tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:53.788074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ab892c-8947-409b-80c0-c003dcd870aa tempest-VolumeQuotaClassesTest-964296390 tempest-VolumeQuotaClassesTest-964296390-project-admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:53.826291 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbeb1e1-8740-4598-a1e3-037c6b204e6e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:53.830331 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbeb1e1-8740-4598-a1e3-037c6b204e6e service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:53.830624 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbeb1e1-8740-4598-a1e3-037c6b204e6e service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:53.830904 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbeb1e1-8740-4598-a1e3-037c6b204e6e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:53.840993 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6dbeb1e1-8740-4598-a1e3-037c6b204e6e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:53.894748 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8d1304-b01c-48d9-9710-4b64eeaeb8c8 service nova] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:53.898256 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7d8d1304-b01c-48d9-9710-4b64eeaeb8c8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:54.058630 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:54.089848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:54.093656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8d1304-b01c-48d9-9710-4b64eeaeb8c8 service nova] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:54.113366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d867fcec-a921-4c68-bc2f-21ff989439bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:54.113366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d867fcec-a921-4c68-bc2f-21ff989439bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:54.113366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d867fcec-a921-4c68-bc2f-21ff989439bc admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:54.113366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d867fcec-a921-4c68-bc2f-21ff989439bc admin admin] PATH_INFO: `/v3/users/04456746d23c484a8196387440affe3e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:54.117915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d867fcec-a921-4c68-bc2f-21ff989439bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=04456746d23c484a8196387440affe3e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:54.132038 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d867fcec-a921-4c68-bc2f-21ff989439bc admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:54.157350 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbeb1e1-8740-4598-a1e3-037c6b204e6e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:54.159953 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbeb1e1-8740-4598-a1e3-037c6b204e6e service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:54.210046 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 700/1416] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:53 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 490 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:54.272996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-d867fcec-a921-4c68-bc2f-21ff989439bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '04456746d23c484a8196387440affe3e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:54.274741 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-d867fcec-a921-4c68-bc2f-21ff989439bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '04456746d23c484a8196387440affe3e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:54.310363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-d867fcec-a921-4c68-bc2f-21ff989439bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '04456746d23c484a8196387440affe3e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:54.337376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-d867fcec-a921-4c68-bc2f-21ff989439bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '04456746d23c484a8196387440affe3e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:54.359069 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-d867fcec-a921-4c68-bc2f-21ff989439bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '04456746d23c484a8196387440affe3e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:54.380370 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 717/1417] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:56:53 2022] DELETE /identity/v3/users/04456746d23c484a8196387440affe3e => generated 0 bytes in 483 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:54.381736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2681fe8e-4dbe-4aeb-9033-c2b34344c99b None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:54.382047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2681fe8e-4dbe-4aeb-9033-c2b34344c99b None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:54.382321 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2681fe8e-4dbe-4aeb-9033-c2b34344c99b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:55.079359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-2681fe8e-4dbe-4aeb-9033-c2b34344c99b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:55.082144 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2681fe8e-4dbe-4aeb-9033-c2b34344c99b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:55.083099 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310c314f-14a4-490e-b7fa-d36a8ee8d457 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:55.083398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310c314f-14a4-490e-b7fa-d36a8ee8d457 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:55.083706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310c314f-14a4-490e-b7fa-d36a8ee8d457 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:55.174866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-310c314f-14a4-490e-b7fa-d36a8ee8d457 None None] MFA Rules not processed for user `14946d97fd384e2a8bff68c4f53c2cfe`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:55.182052 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-310c314f-14a4-490e-b7fa-d36a8ee8d457 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:55.410342 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 701/1418] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 1029 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:55.533378 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 718/1419] 10.222.0.44 () {58 vars in 1015 bytes} [Mon Oct 31 14:56:55 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 451 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:55.575525 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2681fe8e-4dbe-4aeb-9033-c2b34344c99b None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:55.577605 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2681fe8e-4dbe-4aeb-9033-c2b34344c99b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:55.616297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2681fe8e-4dbe-4aeb-9033-c2b34344c99b None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:55.649985 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52670fa-c29f-432b-9938-1bfe51e6daef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:55.650620 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52670fa-c29f-432b-9938-1bfe51e6daef service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:55.650896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52670fa-c29f-432b-9938-1bfe51e6daef service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:55.651179 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52670fa-c29f-432b-9938-1bfe51e6daef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:55.701964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52670fa-c29f-432b-9938-1bfe51e6daef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:55.705833 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52670fa-c29f-432b-9938-1bfe51e6daef service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:55.765135 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 702/1420] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:55.835954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310c314f-14a4-490e-b7fa-d36a8ee8d457 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:55.837509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-310c314f-14a4-490e-b7fa-d36a8ee8d457 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:55.867823 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310c314f-14a4-490e-b7fa-d36a8ee8d457 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:55.888957 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d523cdc-9525-435e-a853-27ea29fff2df service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:55.891032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d523cdc-9525-435e-a853-27ea29fff2df service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:55.892077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d523cdc-9525-435e-a853-27ea29fff2df service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:55.892735 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d523cdc-9525-435e-a853-27ea29fff2df service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:55.956390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d523cdc-9525-435e-a853-27ea29fff2df service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:55.966490 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d523cdc-9525-435e-a853-27ea29fff2df service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:56.056543 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 719/1421] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:56:55 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:56:56.316549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4dfc3e-d665-47ad-9b89-6cdc7c76827c None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:56.317666 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4dfc3e-d665-47ad-9b89-6cdc7c76827c None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:56.318144 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4dfc3e-d665-47ad-9b89-6cdc7c76827c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:56.380286 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-0f4dfc3e-d665-47ad-9b89-6cdc7c76827c None None] MFA Rules not processed for user `75d1901be8064ff5bf79f0268d589e10`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:56.384749 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-0f4dfc3e-d665-47ad-9b89-6cdc7c76827c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:56.678695 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 703/1422] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:56 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 364 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:56:57.288983 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3635f8-ce8b-4a12-a4de-c67a5f857f0a None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:57.289491 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3635f8-ce8b-4a12-a4de-c67a5f857f0a None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:57.289584 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3635f8-ce8b-4a12-a4de-c67a5f857f0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:57.376521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-1b3635f8-ce8b-4a12-a4de-c67a5f857f0a None None] MFA Rules not processed for user `e6bfe2c1ba054935ae8e9fcc054faf68`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:57.389460 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1b3635f8-ce8b-4a12-a4de-c67a5f857f0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:57.622888 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4dfc3e-d665-47ad-9b89-6cdc7c76827c None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:57.624458 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-0f4dfc3e-d665-47ad-9b89-6cdc7c76827c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:57.695062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4dfc3e-d665-47ad-9b89-6cdc7c76827c None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:57.713186 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 720/1423] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:57 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 426 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:56:57.726156 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bcf719-e883-449d-8ba3-39edef8be33d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:57.729244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bcf719-e883-449d-8ba3-39edef8be33d admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:57.738557 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bcf719-e883-449d-8ba3-39edef8be33d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:57.740794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bcf719-e883-449d-8ba3-39edef8be33d admin admin] PATH_INFO: `/v3/projects/32b4f28185d74f9794ce9d7f1feb4445` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:57.744080 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bcf719-e883-449d-8ba3-39edef8be33d admin admin] RBAC: Authorizing `identity:delete_project(project_id=32b4f28185d74f9794ce9d7f1feb4445)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:57.752881 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bcf719-e883-449d-8ba3-39edef8be33d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:58.037886 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c5bcf719-e883-449d-8ba3-39edef8be33d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '32b4f28185d74f9794ce9d7f1feb4445', 'request_id': 'req-c5bcf719-e883-449d-8ba3-39edef8be33d'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:58.049238 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c5bcf719-e883-449d-8ba3-39edef8be33d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '32b4f28185d74f9794ce9d7f1feb4445', 'request_id': 'req-c5bcf719-e883-449d-8ba3-39edef8be33d'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:58.058072 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c5bcf719-e883-449d-8ba3-39edef8be33d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '32b4f28185d74f9794ce9d7f1feb4445', 'request_id': 'req-c5bcf719-e883-449d-8ba3-39edef8be33d'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:58.092565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:58.106164 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c5bcf719-e883-449d-8ba3-39edef8be33d admin admin] The token cache is being invalidate because project 32b4f28185d74f9794ce9d7f1feb4445 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:56:58.106560 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c5bcf719-e883-449d-8ba3-39edef8be33d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:58.110889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c5bcf719-e883-449d-8ba3-39edef8be33d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:58.126666 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 704/1424] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:56:57 2022] DELETE /identity/v3/projects/32b4f28185d74f9794ce9d7f1feb4445 => generated 0 bytes in 504 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:58.130130 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3635f8-ce8b-4a12-a4de-c67a5f857f0a None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:58.150856 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3635f8-ce8b-4a12-a4de-c67a5f857f0a None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:58.230020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:56:58.401193 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8133a72-f316-4416-bf41-8aef371a2bf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:58.404042 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8133a72-f316-4416-bf41-8aef371a2bf3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:58.404969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8133a72-f316-4416-bf41-8aef371a2bf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:58.406198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8133a72-f316-4416-bf41-8aef371a2bf3 admin admin] PATH_INFO: `/v3/users/cffe160549c840ef9b119eaf99111dba` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:58.408611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8133a72-f316-4416-bf41-8aef371a2bf3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cffe160549c840ef9b119eaf99111dba)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:58.412425 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8133a72-f316-4416-bf41-8aef371a2bf3 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:58.563756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b8133a72-f316-4416-bf41-8aef371a2bf3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cffe160549c840ef9b119eaf99111dba'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:58.564921 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b8133a72-f316-4416-bf41-8aef371a2bf3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cffe160549c840ef9b119eaf99111dba'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:58.600080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b8133a72-f316-4416-bf41-8aef371a2bf3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cffe160549c840ef9b119eaf99111dba'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:58.624908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b8133a72-f316-4416-bf41-8aef371a2bf3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cffe160549c840ef9b119eaf99111dba'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:58.642372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b8133a72-f316-4416-bf41-8aef371a2bf3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cffe160549c840ef9b119eaf99111dba'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:58.680844 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 721/1425] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:56:58 2022] DELETE /identity/v3/users/cffe160549c840ef9b119eaf99111dba => generated 0 bytes in 551 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:59.372231 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bcf719-e883-449d-8ba3-39edef8be33d admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:56:59.374430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-c5bcf719-e883-449d-8ba3-39edef8be33d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:56:59.401240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bcf719-e883-449d-8ba3-39edef8be33d admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:56:59.670053 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe31042-f829-4cf0-a802-0a8944629fa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:56:59.672618 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe31042-f829-4cf0-a802-0a8944629fa6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:59.677467 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe31042-f829-4cf0-a802-0a8944629fa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:59.679812 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe31042-f829-4cf0-a802-0a8944629fa6 admin admin] PATH_INFO: `/v3/users/7b4a8dfd5e5441eb881b1320d54f9bf4` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:59.683126 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe31042-f829-4cf0-a802-0a8944629fa6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b4a8dfd5e5441eb881b1320d54f9bf4)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:56:59.690291 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe31042-f829-4cf0-a802-0a8944629fa6 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:56:59.841724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-4fe31042-f829-4cf0-a802-0a8944629fa6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b4a8dfd5e5441eb881b1320d54f9bf4'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:59.862557 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-4fe31042-f829-4cf0-a802-0a8944629fa6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b4a8dfd5e5441eb881b1320d54f9bf4'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:59.863628 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-4fe31042-f829-4cf0-a802-0a8944629fa6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b4a8dfd5e5441eb881b1320d54f9bf4'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:59.864565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-4fe31042-f829-4cf0-a802-0a8944629fa6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b4a8dfd5e5441eb881b1320d54f9bf4'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:59.897542 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-4fe31042-f829-4cf0-a802-0a8944629fa6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b4a8dfd5e5441eb881b1320d54f9bf4'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:56:59.925017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 705/1426] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:56:59 2022] DELETE /identity/v3/users/7b4a8dfd5e5441eb881b1320d54f9bf4 => generated 0 bytes in 551 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:56:59.926969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbb16fa-f4ae-4947-a10f-188906f9648b None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:56:59.927468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbb16fa-f4ae-4947-a10f-188906f9648b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:56:59.927661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbb16fa-f4ae-4947-a10f-188906f9648b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:56:59.978969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-dfbb16fa-f4ae-4947-a10f-188906f9648b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:56:59.982280 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-dfbb16fa-f4ae-4947-a10f-188906f9648b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:00.232832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 722/1427] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:56:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:00.349110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe31042-f829-4cf0-a802-0a8944629fa6 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:00.351163 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-4fe31042-f829-4cf0-a802-0a8944629fa6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:00.502441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe31042-f829-4cf0-a802-0a8944629fa6 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:00.518084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493054ed-b9b7-4dd9-9a6e-257fe13215ff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:00.518916 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493054ed-b9b7-4dd9-9a6e-257fe13215ff service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:00.519204 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493054ed-b9b7-4dd9-9a6e-257fe13215ff service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:00.523902 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493054ed-b9b7-4dd9-9a6e-257fe13215ff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:00.581442 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493054ed-b9b7-4dd9-9a6e-257fe13215ff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:00.593882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493054ed-b9b7-4dd9-9a6e-257fe13215ff service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:00.646998 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 706/1428] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:02.205963 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbb16fa-f4ae-4947-a10f-188906f9648b None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:02.208520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-dfbb16fa-f4ae-4947-a10f-188906f9648b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:02.238003 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbb16fa-f4ae-4947-a10f-188906f9648b None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:02.252333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa58e54-8f20-424b-93cd-a871cfcdf238 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:02.252993 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa58e54-8f20-424b-93cd-a871cfcdf238 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:02.253275 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa58e54-8f20-424b-93cd-a871cfcdf238 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:02.258679 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa58e54-8f20-424b-93cd-a871cfcdf238 admin admin] PATH_INFO: `/v3/projects/acd1ab2e91a74fd089e1a6c539e7961a` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:02.260995 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa58e54-8f20-424b-93cd-a871cfcdf238 admin admin] RBAC: Authorizing `identity:delete_project(project_id=acd1ab2e91a74fd089e1a6c539e7961a)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:02.264870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa58e54-8f20-424b-93cd-a871cfcdf238 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:02.377906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:02.430386 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-eaa58e54-8f20-424b-93cd-a871cfcdf238 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'acd1ab2e91a74fd089e1a6c539e7961a', 'request_id': 'req-eaa58e54-8f20-424b-93cd-a871cfcdf238'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:02.453483 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-eaa58e54-8f20-424b-93cd-a871cfcdf238 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'acd1ab2e91a74fd089e1a6c539e7961a', 'request_id': 'req-eaa58e54-8f20-424b-93cd-a871cfcdf238'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:02.476757 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-eaa58e54-8f20-424b-93cd-a871cfcdf238 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'acd1ab2e91a74fd089e1a6c539e7961a', 'request_id': 'req-eaa58e54-8f20-424b-93cd-a871cfcdf238'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:02.514994 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-eaa58e54-8f20-424b-93cd-a871cfcdf238 admin admin] The token cache is being invalidate because project acd1ab2e91a74fd089e1a6c539e7961a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:02.516555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-eaa58e54-8f20-424b-93cd-a871cfcdf238 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:02.519688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-eaa58e54-8f20-424b-93cd-a871cfcdf238 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:02.525731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 723/1429] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:02 2022] DELETE /identity/v3/projects/acd1ab2e91a74fd089e1a6c539e7961a => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:02.531826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88779869-549d-4364-828f-be27123db5b0 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:02.532713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88779869-549d-4364-828f-be27123db5b0 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:02.533297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88779869-549d-4364-828f-be27123db5b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:02.595895 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-88779869-549d-4364-828f-be27123db5b0 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:02.615018 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-88779869-549d-4364-828f-be27123db5b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:02.685399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:02.893342 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 707/1430] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 362 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:02.896933 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f197b236-aae7-488a-a12f-bc1176467c31 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:02.897498 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f197b236-aae7-488a-a12f-bc1176467c31 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:02.898276 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f197b236-aae7-488a-a12f-bc1176467c31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:02.950922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-f197b236-aae7-488a-a12f-bc1176467c31 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:02.967037 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-f197b236-aae7-488a-a12f-bc1176467c31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:02.981383 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839c6e03-05ef-4c64-a5c7-34a119afd2d1 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:02.981676 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839c6e03-05ef-4c64-a5c7-34a119afd2d1 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:02.982623 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839c6e03-05ef-4c64-a5c7-34a119afd2d1 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:02.984217 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 708/1431] 10.222.0.36 () {54 vars in 921 bytes} [Mon Oct 31 14:57:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:57:03.004559 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 724/1432] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:03.006063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839c6e03-05ef-4c64-a5c7-34a119afd2d1 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:03.031966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839c6e03-05ef-4c64-a5c7-34a119afd2d1 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:03.048062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3e8200-c9a6-4ebf-b783-73da21613818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:03.049999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3e8200-c9a6-4ebf-b783-73da21613818 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:03.062023 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3e8200-c9a6-4ebf-b783-73da21613818 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:03.062383 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3e8200-c9a6-4ebf-b783-73da21613818 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:03.068827 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3e8200-c9a6-4ebf-b783-73da21613818 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:57:03.075625 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3e8200-c9a6-4ebf-b783-73da21613818 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:03.078782 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3e8200-c9a6-4ebf-b783-73da21613818 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:03.141945 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 709/1433] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:57:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:57:03.144690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f197b236-aae7-488a-a12f-bc1176467c31 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:03.156699 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f197b236-aae7-488a-a12f-bc1176467c31 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:03.189321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858807f7-4e04-4cc8-ba05-8dee2c1493f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:03.196407 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858807f7-4e04-4cc8-ba05-8dee2c1493f4 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:03.196701 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858807f7-4e04-4cc8-ba05-8dee2c1493f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:03.196963 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858807f7-4e04-4cc8-ba05-8dee2c1493f4 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:03.197639 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858807f7-4e04-4cc8-ba05-8dee2c1493f4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:03.199641 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858807f7-4e04-4cc8-ba05-8dee2c1493f4 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:03.204772 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d3f217-7e1a-48cc-8f76-f8b7b25dde64 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:03.205179 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d3f217-7e1a-48cc-8f76-f8b7b25dde64 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:03.205297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d3f217-7e1a-48cc-8f76-f8b7b25dde64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:03.289815 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-d7d3f217-7e1a-48cc-8f76-f8b7b25dde64 None None] MFA Rules not processed for user `c8cace029a92406f850bb0a4ca07c461`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:03.291053 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d7d3f217-7e1a-48cc-8f76-f8b7b25dde64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:03.376336 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 725/1434] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:03 2022] POST /identity/v3/projects => generated 385 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:57:03.386455 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858807f7-4e04-4cc8-ba05-8dee2c1493f4 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:03.641348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 710/1435] 10.222.0.36 () {58 vars in 1018 bytes} [Mon Oct 31 14:57:03 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 438 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:57:03.706696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858807f7-4e04-4cc8-ba05-8dee2c1493f4 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:03.729842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1200a1-d3d2-4da2-b91d-f770fb30e1c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:03.732828 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1200a1-d3d2-4da2-b91d-f770fb30e1c2 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:03.733805 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1200a1-d3d2-4da2-b91d-f770fb30e1c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:03.738289 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1200a1-d3d2-4da2-b91d-f770fb30e1c2 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:03.739371 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1200a1-d3d2-4da2-b91d-f770fb30e1c2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:03.745190 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1200a1-d3d2-4da2-b91d-f770fb30e1c2 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:03.803144 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 726/1436] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:03 2022] POST /identity/v3/users => generated 293 bytes in 418 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:03.807194 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d3f217-7e1a-48cc-8f76-f8b7b25dde64 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:03.838464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d3f217-7e1a-48cc-8f76-f8b7b25dde64 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:03.869761 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936d6902-85d9-4da0-8629-441173fd1504 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:03.870416 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d6902-85d9-4da0-8629-441173fd1504 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:03.870699 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d6902-85d9-4da0-8629-441173fd1504 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:03.870972 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d6902-85d9-4da0-8629-441173fd1504 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:03.872888 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936d6902-85d9-4da0-8629-441173fd1504 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:03.874563 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936d6902-85d9-4da0-8629-441173fd1504 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:03.901011 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1200a1-d3d2-4da2-b91d-f770fb30e1c2 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:03.920066 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1200a1-d3d2-4da2-b91d-f770fb30e1c2 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:03.920755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 711/1437] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:03 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:03.940368 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c45719a-220c-4a51-ba89-f2adee41bfc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:03.941050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c45719a-220c-4a51-ba89-f2adee41bfc5 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:03.941448 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c45719a-220c-4a51-ba89-f2adee41bfc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:03.941595 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c45719a-220c-4a51-ba89-f2adee41bfc5 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:03.942251 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c45719a-220c-4a51-ba89-f2adee41bfc5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:03.944096 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c45719a-220c-4a51-ba89-f2adee41bfc5 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:03.978326 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 727/1438] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:03 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 14:57:03.983265 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936d6902-85d9-4da0-8629-441173fd1504 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:03.998920 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936d6902-85d9-4da0-8629-441173fd1504 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:04.013366 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b87fcb1-a547-4673-96ff-0574b31edd6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:04.014061 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b87fcb1-a547-4673-96ff-0574b31edd6b admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:04.014349 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b87fcb1-a547-4673-96ff-0574b31edd6b admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:04.014628 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b87fcb1-a547-4673-96ff-0574b31edd6b admin admin] PATH_INFO: `/v3/projects/fbcaed4971c04c66b46ee6b90d6b54be/users/9d564924dc2347fc9aff513353941d84/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:04.050411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b87fcb1-a547-4673-96ff-0574b31edd6b admin admin] RBAC: Authorizing `identity:create_grant(project_id=fbcaed4971c04c66b46ee6b90d6b54be, user_id=9d564924dc2347fc9aff513353941d84, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:04.053175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b87fcb1-a547-4673-96ff-0574b31edd6b admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:04.085171 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 712/1439] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:03 2022] PUT /identity/v3/projects/fbcaed4971c04c66b46ee6b90d6b54be/users/9d564924dc2347fc9aff513353941d84/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:04.096205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24c22b2-08d1-4d86-b61b-3d911fa50b16 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:04.096523 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24c22b2-08d1-4d86-b61b-3d911fa50b16 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:04.096804 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24c22b2-08d1-4d86-b61b-3d911fa50b16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:04.193978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-a24c22b2-08d1-4d86-b61b-3d911fa50b16 None None] MFA Rules not processed for user `9d564924dc2347fc9aff513353941d84`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:04.195744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a24c22b2-08d1-4d86-b61b-3d911fa50b16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:04.288663 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:04.303781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 728/1440] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:04 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:04.308095 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b87fcb1-a547-4673-96ff-0574b31edd6b admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:04.317312 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24c22b2-08d1-4d86-b61b-3d911fa50b16 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:04.323818 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a24c22b2-08d1-4d86-b61b-3d911fa50b16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:04.491445 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b87fcb1-a547-4673-96ff-0574b31edd6b admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:04.499341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b2e79f-1621-4f6b-9236-17ad86b8dc11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:04.500253 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b2e79f-1621-4f6b-9236-17ad86b8dc11 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:04.500744 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b2e79f-1621-4f6b-9236-17ad86b8dc11 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:04.501250 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b2e79f-1621-4f6b-9236-17ad86b8dc11 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:04.502251 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b2e79f-1621-4f6b-9236-17ad86b8dc11 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:04.505337 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b2e79f-1621-4f6b-9236-17ad86b8dc11 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:04.538110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:04.573326 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24c22b2-08d1-4d86-b61b-3d911fa50b16 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:04.585687 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40497b68-35a3-48c5-b1e6-b54c27eb3cca service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:04.589047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40497b68-35a3-48c5-b1e6-b54c27eb3cca service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:04.593032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40497b68-35a3-48c5-b1e6-b54c27eb3cca service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:04.593328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40497b68-35a3-48c5-b1e6-b54c27eb3cca service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:04.598443 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 713/1441] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:04 2022] POST /identity/v3/projects => generated 387 bytes in 291 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:04.601054 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b2e79f-1621-4f6b-9236-17ad86b8dc11 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:04.604863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-19b2e79f-1621-4f6b-9236-17ad86b8dc11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:04.628204 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40497b68-35a3-48c5-b1e6-b54c27eb3cca service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:04.631235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40497b68-35a3-48c5-b1e6-b54c27eb3cca service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:04.672176 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 729/1442] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:04.676600 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40497b68-35a3-48c5-b1e6-b54c27eb3cca service nova] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:04.731247 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40497b68-35a3-48c5-b1e6-b54c27eb3cca service nova] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:04.880060 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b2e79f-1621-4f6b-9236-17ad86b8dc11 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:04.897075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:04.900090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:04.903493 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:04.903854 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b admin admin] PATH_INFO: `/v3/projects/a95e80610b6f49cb85c7b2f7706056d2` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:04.907812 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b admin admin] RBAC: Authorizing `identity:delete_project(project_id=a95e80610b6f49cb85c7b2f7706056d2)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:04.914309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:04.953251 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729b16f3-c2e1-4a44-8852-71fe1dca4bdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:04.953846 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729b16f3-c2e1-4a44-8852-71fe1dca4bdb admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:04.955409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729b16f3-c2e1-4a44-8852-71fe1dca4bdb admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:04.955726 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729b16f3-c2e1-4a44-8852-71fe1dca4bdb admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:04.956381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729b16f3-c2e1-4a44-8852-71fe1dca4bdb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:04.965504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729b16f3-c2e1-4a44-8852-71fe1dca4bdb admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:05.029047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 730/1443] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:04 2022] POST /identity/v3/users => generated 294 bytes in 356 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:05.031255 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729b16f3-c2e1-4a44-8852-71fe1dca4bdb admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:05.054966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a95e80610b6f49cb85c7b2f7706056d2', 'request_id': 'req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:05.068487 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729b16f3-c2e1-4a44-8852-71fe1dca4bdb admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:05.081896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a95e80610b6f49cb85c7b2f7706056d2', 'request_id': 'req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:05.106084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a95e80610b6f49cb85c7b2f7706056d2', 'request_id': 'req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:05.193692 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b admin admin] The token cache is being invalidate because project a95e80610b6f49cb85c7b2f7706056d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:05.194050 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:05.194829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:05.198748 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 714/1444] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:04 2022] DELETE /identity/v3/projects/a95e80610b6f49cb85c7b2f7706056d2 => generated 0 bytes in 600 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:05.203191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:05.204964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:05.437855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3a1954-3378-4ef2-8c93-c034c2d0bce9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:05.450793 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88cc814-5956-4c52-b86a-3c48e0fe6e2b admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:05.455182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3a1954-3378-4ef2-8c93-c034c2d0bce9 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:05.455496 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3a1954-3378-4ef2-8c93-c034c2d0bce9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:05.455797 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3a1954-3378-4ef2-8c93-c034c2d0bce9 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:05.456541 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3a1954-3378-4ef2-8c93-c034c2d0bce9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:05.458264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3a1954-3378-4ef2-8c93-c034c2d0bce9 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:05.470309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572391bf-fba7-419c-b69c-85119c14bf7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:05.471443 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572391bf-fba7-419c-b69c-85119c14bf7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:05.472432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572391bf-fba7-419c-b69c-85119c14bf7b admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:05.473071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572391bf-fba7-419c-b69c-85119c14bf7b admin admin] PATH_INFO: `/v3/users/01f207b30ff24fb2b25ae6c4fb7d8dde` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:05.475096 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572391bf-fba7-419c-b69c-85119c14bf7b admin admin] RBAC: Authorizing `identity:delete_user(user_id=01f207b30ff24fb2b25ae6c4fb7d8dde)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:05.491707 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572391bf-fba7-419c-b69c-85119c14bf7b admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:05.493833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 731/1445] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:05 2022] GET /identity/v3/roles => generated 1420 bytes in 463 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:05.497419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3a1954-3378-4ef2-8c93-c034c2d0bce9 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:05.513166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-4c3a1954-3378-4ef2-8c93-c034c2d0bce9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:05.555356 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3a1954-3378-4ef2-8c93-c034c2d0bce9 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:05.575082 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:05.579349 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82bb02b-7a1f-47b9-8577-377791d9634e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:05.583400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82bb02b-7a1f-47b9-8577-377791d9634e admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:05.583992 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82bb02b-7a1f-47b9-8577-377791d9634e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:05.584263 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82bb02b-7a1f-47b9-8577-377791d9634e admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:05.584938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82bb02b-7a1f-47b9-8577-377791d9634e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:05.595676 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82bb02b-7a1f-47b9-8577-377791d9634e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:05.601969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:05.639701 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 732/1446] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:05 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:05.641864 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82bb02b-7a1f-47b9-8577-377791d9634e admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:05.667415 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-572391bf-fba7-419c-b69c-85119c14bf7b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '01f207b30ff24fb2b25ae6c4fb7d8dde'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:05.691003 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82bb02b-7a1f-47b9-8577-377791d9634e admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:05.692695 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-572391bf-fba7-419c-b69c-85119c14bf7b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '01f207b30ff24fb2b25ae6c4fb7d8dde'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:05.695452 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-572391bf-fba7-419c-b69c-85119c14bf7b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '01f207b30ff24fb2b25ae6c4fb7d8dde'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:05.697244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-572391bf-fba7-419c-b69c-85119c14bf7b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '01f207b30ff24fb2b25ae6c4fb7d8dde'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:05.719526 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196e0db0-95a7-4d81-a278-d7544434497b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:05.720574 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196e0db0-95a7-4d81-a278-d7544434497b admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:05.720970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196e0db0-95a7-4d81-a278-d7544434497b admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:05.721409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196e0db0-95a7-4d81-a278-d7544434497b admin admin] PATH_INFO: `/v3/projects/78d7dc6835e64bc19dc66cb00c3f47ee/users/c6cabcb2eb45410199ed41806ffa9958/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:05.731096 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-572391bf-fba7-419c-b69c-85119c14bf7b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '01f207b30ff24fb2b25ae6c4fb7d8dde'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:05.801055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196e0db0-95a7-4d81-a278-d7544434497b admin admin] RBAC: Authorizing `identity:create_grant(project_id=78d7dc6835e64bc19dc66cb00c3f47ee, user_id=c6cabcb2eb45410199ed41806ffa9958, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:05.808152 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196e0db0-95a7-4d81-a278-d7544434497b admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:05.809421 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 715/1447] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:05 2022] DELETE /identity/v3/users/01f207b30ff24fb2b25ae6c4fb7d8dde => generated 0 bytes in 553 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:05.847827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 733/1448] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:05 2022] PUT /identity/v3/projects/78d7dc6835e64bc19dc66cb00c3f47ee/users/c6cabcb2eb45410199ed41806ffa9958/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:05.861107 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3cd3f3-cdf8-4676-bd04-3572f694034f None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:05.862076 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3cd3f3-cdf8-4676-bd04-3572f694034f None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:05.862376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3cd3f3-cdf8-4676-bd04-3572f694034f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:05.993106 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-1d3cd3f3-cdf8-4676-bd04-3572f694034f None None] MFA Rules not processed for user `c6cabcb2eb45410199ed41806ffa9958`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:05.994527 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-1d3cd3f3-cdf8-4676-bd04-3572f694034f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:06.107466 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf67339-8219-409b-9363-6322628c67b0 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:06.111535 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf67339-8219-409b-9363-6322628c67b0 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:06.111535 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf67339-8219-409b-9363-6322628c67b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:06.144289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 716/1449] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:05 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:06.147849 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3cd3f3-cdf8-4676-bd04-3572f694034f None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:06.149447 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-1d3cd3f3-cdf8-4676-bd04-3572f694034f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:06.165660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-fdf67339-8219-409b-9363-6322628c67b0 None None] MFA Rules not processed for user `cdcbcb2ed9e44dbe8444e531fc1eb707`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:06.166987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-fdf67339-8219-409b-9363-6322628c67b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:06.334645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 734/1450] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:06 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:06.378715 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3cd3f3-cdf8-4676-bd04-3572f694034f None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:06.394167 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc9b086-b273-4031-bf58-7fc69bdaddf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:06.396477 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc9b086-b273-4031-bf58-7fc69bdaddf1 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:06.397894 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc9b086-b273-4031-bf58-7fc69bdaddf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:06.398429 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc9b086-b273-4031-bf58-7fc69bdaddf1 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:06.400721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc9b086-b273-4031-bf58-7fc69bdaddf1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:06.409777 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc9b086-b273-4031-bf58-7fc69bdaddf1 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:06.483984 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 717/1451] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:06 2022] POST /identity/v3/projects => generated 387 bytes in 336 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:06.485871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf67339-8219-409b-9363-6322628c67b0 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:06.503360 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf67339-8219-409b-9363-6322628c67b0 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:06.670215 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312ab041-99b4-4b6e-9193-bcf28af1e6dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:06.671146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312ab041-99b4-4b6e-9193-bcf28af1e6dd admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:06.671385 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312ab041-99b4-4b6e-9193-bcf28af1e6dd admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:06.671750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312ab041-99b4-4b6e-9193-bcf28af1e6dd admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:06.672680 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312ab041-99b4-4b6e-9193-bcf28af1e6dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:06.676836 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312ab041-99b4-4b6e-9193-bcf28af1e6dd admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:06.713119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 735/1452] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:06 2022] POST /identity/v3/users => generated 300 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:06.716880 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc9b086-b273-4031-bf58-7fc69bdaddf1 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:06.728314 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc9b086-b273-4031-bf58-7fc69bdaddf1 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:06.743865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd31b8a-6d0e-4ee5-b6c7-83d9d78af5f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:06.744533 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd31b8a-6d0e-4ee5-b6c7-83d9d78af5f2 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:06.744842 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd31b8a-6d0e-4ee5-b6c7-83d9d78af5f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:06.745117 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd31b8a-6d0e-4ee5-b6c7-83d9d78af5f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:06.745781 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd31b8a-6d0e-4ee5-b6c7-83d9d78af5f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:06.754858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd31b8a-6d0e-4ee5-b6c7-83d9d78af5f2 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:06.774050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312ab041-99b4-4b6e-9193-bcf28af1e6dd admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:06.781064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 718/1453] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:06 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:06.785594 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312ab041-99b4-4b6e-9193-bcf28af1e6dd admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:06.791970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab2f451-0a33-445d-ab91-942cd89fe067 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:06.792770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab2f451-0a33-445d-ab91-942cd89fe067 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:06.793361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab2f451-0a33-445d-ab91-942cd89fe067 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:06.793743 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab2f451-0a33-445d-ab91-942cd89fe067 admin admin] PATH_INFO: `/v3/projects/d21354c3e7ae481dbd38c5e3deead403/users/4c68892fdcb34d0cbec882eba28b097a/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:06.836497 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab2f451-0a33-445d-ab91-942cd89fe067 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d21354c3e7ae481dbd38c5e3deead403, user_id=4c68892fdcb34d0cbec882eba28b097a, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:06.844783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab2f451-0a33-445d-ab91-942cd89fe067 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:06.883571 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 736/1454] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:06 2022] PUT /identity/v3/projects/d21354c3e7ae481dbd38c5e3deead403/users/4c68892fdcb34d0cbec882eba28b097a/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:06.887793 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd31b8a-6d0e-4ee5-b6c7-83d9d78af5f2 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:06.901719 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=92,keystone:DELETE=6,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:06.906274 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd31b8a-6d0e-4ee5-b6c7-83d9d78af5f2 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:07.040191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b73697e-d3cd-4a42-a2f9-68d61c75e34a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:07.044737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b73697e-d3cd-4a42-a2f9-68d61c75e34a admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:07.045023 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b73697e-d3cd-4a42-a2f9-68d61c75e34a admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:07.045303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b73697e-d3cd-4a42-a2f9-68d61c75e34a admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:07.053898 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b73697e-d3cd-4a42-a2f9-68d61c75e34a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:07.054374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b73697e-d3cd-4a42-a2f9-68d61c75e34a admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:07.075854 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 719/1455] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:06 2022] GET /identity/v3/roles => generated 1420 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:07.080243 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab2f451-0a33-445d-ab91-942cd89fe067 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:07.096943 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab2f451-0a33-445d-ab91-942cd89fe067 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:07.111366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0467ca4-25fb-4d4d-856e-1d86d709e3f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:07.112099 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0467ca4-25fb-4d4d-856e-1d86d709e3f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:07.112384 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0467ca4-25fb-4d4d-856e-1d86d709e3f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:07.112660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0467ca4-25fb-4d4d-856e-1d86d709e3f2 admin admin] PATH_INFO: `/v3/projects/d21354c3e7ae481dbd38c5e3deead403/users/4c68892fdcb34d0cbec882eba28b097a/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:07.114690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0467ca4-25fb-4d4d-856e-1d86d709e3f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d21354c3e7ae481dbd38c5e3deead403, user_id=4c68892fdcb34d0cbec882eba28b097a, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:07.118575 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0467ca4-25fb-4d4d-856e-1d86d709e3f2 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:07.142663 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b73697e-d3cd-4a42-a2f9-68d61c75e34a admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:07.145560 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-1b73697e-d3cd-4a42-a2f9-68d61c75e34a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:07.167586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 737/1456] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:07 2022] PUT /identity/v3/projects/d21354c3e7ae481dbd38c5e3deead403/users/4c68892fdcb34d0cbec882eba28b097a/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:07.170364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b58ccee-eb6b-4f4f-a03c-93982aa49b6f None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:07.170673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b58ccee-eb6b-4f4f-a03c-93982aa49b6f None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:07.170933 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b58ccee-eb6b-4f4f-a03c-93982aa49b6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:07.181904 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b73697e-d3cd-4a42-a2f9-68d61c75e34a admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:07.245424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-7b58ccee-eb6b-4f4f-a03c-93982aa49b6f None None] MFA Rules not processed for user `4c68892fdcb34d0cbec882eba28b097a`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:07.247128 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7b58ccee-eb6b-4f4f-a03c-93982aa49b6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:07.296489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:07.358312 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78fc6747-acab-4d27-b5ca-f04bc33ea12e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:07.358979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fc6747-acab-4d27-b5ca-f04bc33ea12e admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:07.359285 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fc6747-acab-4d27-b5ca-f04bc33ea12e admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:07.359578 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fc6747-acab-4d27-b5ca-f04bc33ea12e admin admin] PATH_INFO: `/v3/projects/f9de63c41ca84694b34482bf0d56e08d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:07.368226 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fc6747-acab-4d27-b5ca-f04bc33ea12e admin admin] RBAC: Authorizing `identity:delete_project(project_id=f9de63c41ca84694b34482bf0d56e08d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:07.373598 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fc6747-acab-4d27-b5ca-f04bc33ea12e admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:07.439970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 738/1457] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:07 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:07.453529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b58ccee-eb6b-4f4f-a03c-93982aa49b6f None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:07.458059 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7b58ccee-eb6b-4f4f-a03c-93982aa49b6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:07.536533 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-78fc6747-acab-4d27-b5ca-f04bc33ea12e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f9de63c41ca84694b34482bf0d56e08d', 'request_id': 'req-78fc6747-acab-4d27-b5ca-f04bc33ea12e'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:07.561430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-78fc6747-acab-4d27-b5ca-f04bc33ea12e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f9de63c41ca84694b34482bf0d56e08d', 'request_id': 'req-78fc6747-acab-4d27-b5ca-f04bc33ea12e'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:07.568821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-78fc6747-acab-4d27-b5ca-f04bc33ea12e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f9de63c41ca84694b34482bf0d56e08d', 'request_id': 'req-78fc6747-acab-4d27-b5ca-f04bc33ea12e'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:07.604976 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-78fc6747-acab-4d27-b5ca-f04bc33ea12e admin admin] The token cache is being invalidate because project f9de63c41ca84694b34482bf0d56e08d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:07.605924 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-78fc6747-acab-4d27-b5ca-f04bc33ea12e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:07.611306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-78fc6747-acab-4d27-b5ca-f04bc33ea12e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:07.614568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 720/1458] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:07 2022] DELETE /identity/v3/projects/f9de63c41ca84694b34482bf0d56e08d => generated 0 bytes in 472 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:07.621222 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74ca603-c773-4254-af9a-37fcb406d08a None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:07.621817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74ca603-c773-4254-af9a-37fcb406d08a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:07.622039 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74ca603-c773-4254-af9a-37fcb406d08a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:07.640847 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b58ccee-eb6b-4f4f-a03c-93982aa49b6f None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:07.650106 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-a74ca603-c773-4254-af9a-37fcb406d08a None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:07.651731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a74ca603-c773-4254-af9a-37fcb406d08a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:07.709776 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:07.748323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab0d3ec-7940-4bb9-94e2-8666c8348755 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:07.748983 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab0d3ec-7940-4bb9-94e2-8666c8348755 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:07.749269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab0d3ec-7940-4bb9-94e2-8666c8348755 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:07.750050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab0d3ec-7940-4bb9-94e2-8666c8348755 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:07.752576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2ab0d3ec-7940-4bb9-94e2-8666c8348755 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:07.801250 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 721/1459] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:57:07.810323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae8c4ba-ad0d-4870-89fa-ac6dcdecf825 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:07.810883 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae8c4ba-ad0d-4870-89fa-ac6dcdecf825 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:07.811250 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae8c4ba-ad0d-4870-89fa-ac6dcdecf825 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:07.858359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-5ae8c4ba-ad0d-4870-89fa-ac6dcdecf825 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:07.860618 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5ae8c4ba-ad0d-4870-89fa-ac6dcdecf825 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:07.865570 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab0d3ec-7940-4bb9-94e2-8666c8348755 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:07.869075 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab0d3ec-7940-4bb9-94e2-8666c8348755 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:07.888448 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 722/1460] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:57:07.893083 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae8c4ba-ad0d-4870-89fa-ac6dcdecf825 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:07.896437 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 739/1461] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:07 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:07.906571 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae8c4ba-ad0d-4870-89fa-ac6dcdecf825 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:07.928289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7b8321-7f2c-4726-b55c-9360261a2584 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:07.929033 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7b8321-7f2c-4726-b55c-9360261a2584 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:07.931889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7b8321-7f2c-4726-b55c-9360261a2584 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:07.932262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7b8321-7f2c-4726-b55c-9360261a2584 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:07.933145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7b8321-7f2c-4726-b55c-9360261a2584 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:57:07.933598 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7b8321-7f2c-4726-b55c-9360261a2584 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:07.935682 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7b8321-7f2c-4726-b55c-9360261a2584 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:07.982646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 723/1462] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:57:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:57:07.983443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab0d3ec-7940-4bb9-94e2-8666c8348755 service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:08.002849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab0d3ec-7940-4bb9-94e2-8666c8348755 service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:08.020957 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04556b99-2824-4d29-a376-4b91adde2129 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:08.021601 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04556b99-2824-4d29-a376-4b91adde2129 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:08.021913 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04556b99-2824-4d29-a376-4b91adde2129 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:08.022154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04556b99-2824-4d29-a376-4b91adde2129 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:08.022807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04556b99-2824-4d29-a376-4b91adde2129 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:08.024778 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04556b99-2824-4d29-a376-4b91adde2129 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:08.056897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:08.092239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7b8321-7f2c-4726-b55c-9360261a2584 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:08.092782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 740/1463] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:07 2022] POST /identity/v3/projects => generated 387 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:08.111410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7b8321-7f2c-4726-b55c-9360261a2584 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:08.274358 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6250e6a-a346-4d02-a3ff-996872622332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:08.275477 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6250e6a-a346-4d02-a3ff-996872622332 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:08.276467 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6250e6a-a346-4d02-a3ff-996872622332 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:08.277390 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6250e6a-a346-4d02-a3ff-996872622332 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:08.279690 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6250e6a-a346-4d02-a3ff-996872622332 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:08.282319 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6250e6a-a346-4d02-a3ff-996872622332 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:08.347888 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 724/1464] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:08 2022] POST /identity/v3/users => generated 294 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:08.350481 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04556b99-2824-4d29-a376-4b91adde2129 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:08.368992 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04556b99-2824-4d29-a376-4b91adde2129 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:08.378349 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e1ae8f-b5f4-490f-8bb8-cfb420c1f1f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:08.379413 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e1ae8f-b5f4-490f-8bb8-cfb420c1f1f6 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:08.379413 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e1ae8f-b5f4-490f-8bb8-cfb420c1f1f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:08.379763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e1ae8f-b5f4-490f-8bb8-cfb420c1f1f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:08.380306 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e1ae8f-b5f4-490f-8bb8-cfb420c1f1f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:08.381904 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e1ae8f-b5f4-490f-8bb8-cfb420c1f1f6 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:08.410578 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 741/1465] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:08 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:08.414345 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6250e6a-a346-4d02-a3ff-996872622332 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:08.431162 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6250e6a-a346-4d02-a3ff-996872622332 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:08.449586 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eacf2197-fd84-4f72-99a0-055c2258a35c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:08.450352 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacf2197-fd84-4f72-99a0-055c2258a35c admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:08.451497 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacf2197-fd84-4f72-99a0-055c2258a35c admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:08.451497 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacf2197-fd84-4f72-99a0-055c2258a35c admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:08.451797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eacf2197-fd84-4f72-99a0-055c2258a35c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:08.453552 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eacf2197-fd84-4f72-99a0-055c2258a35c admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:08.491424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 725/1466] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:08 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:08.491668 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e1ae8f-b5f4-490f-8bb8-cfb420c1f1f6 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:08.507871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e1ae8f-b5f4-490f-8bb8-cfb420c1f1f6 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:08.523647 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62ba641-c107-41c8-92ff-10ac24ee8922 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:08.524032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62ba641-c107-41c8-92ff-10ac24ee8922 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:08.524390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62ba641-c107-41c8-92ff-10ac24ee8922 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:08.524776 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62ba641-c107-41c8-92ff-10ac24ee8922 admin admin] PATH_INFO: `/v3/projects/966e35ad3ebe4c969cc5bba3bef68c57/users/b00c33424f954948af34cae0907f786c/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:08.578444 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62ba641-c107-41c8-92ff-10ac24ee8922 admin admin] RBAC: Authorizing `identity:create_grant(project_id=966e35ad3ebe4c969cc5bba3bef68c57, user_id=b00c33424f954948af34cae0907f786c, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:08.585995 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62ba641-c107-41c8-92ff-10ac24ee8922 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:08.622857 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 742/1467] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:08 2022] PUT /identity/v3/projects/966e35ad3ebe4c969cc5bba3bef68c57/users/b00c33424f954948af34cae0907f786c/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:08.633623 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d52c0-767a-4349-bcd4-8d72a00c6658 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:08.633879 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d52c0-767a-4349-bcd4-8d72a00c6658 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:08.634206 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d52c0-767a-4349-bcd4-8d72a00c6658 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:08.689023 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-ba2d52c0-767a-4349-bcd4-8d72a00c6658 None None] MFA Rules not processed for user `b00c33424f954948af34cae0907f786c`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:08.690310 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ba2d52c0-767a-4349-bcd4-8d72a00c6658 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:08.796748 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 726/1468] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:08 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:57:08.797229 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62ba641-c107-41c8-92ff-10ac24ee8922 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:09.052614 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62ba641-c107-41c8-92ff-10ac24ee8922 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:09.070031 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e56e47c-92af-4fa8-ae8c-cf57f47a9091 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:09.070669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e56e47c-92af-4fa8-ae8c-cf57f47a9091 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:09.070949 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e56e47c-92af-4fa8-ae8c-cf57f47a9091 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:09.071213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2d52c0-767a-4349-bcd4-8d72a00c6658 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:09.071343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e56e47c-92af-4fa8-ae8c-cf57f47a9091 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:09.071950 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e56e47c-92af-4fa8-ae8c-cf57f47a9091 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:09.073883 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e56e47c-92af-4fa8-ae8c-cf57f47a9091 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:09.074180 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ba2d52c0-767a-4349-bcd4-8d72a00c6658 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:09.129893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:09.160786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 743/1469] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:08 2022] POST /identity/v3/projects => generated 387 bytes in 364 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:09.161739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e56e47c-92af-4fa8-ae8c-cf57f47a9091 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:09.240718 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2d52c0-767a-4349-bcd4-8d72a00c6658 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:09.361345 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e56e47c-92af-4fa8-ae8c-cf57f47a9091 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:09.371220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657b8991-7696-4903-956e-b7146eadca40 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:09.373198 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657b8991-7696-4903-956e-b7146eadca40 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:09.373524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657b8991-7696-4903-956e-b7146eadca40 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:09.376078 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657b8991-7696-4903-956e-b7146eadca40 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:09.376942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61a2259-b0b0-453b-8b22-07c062919031 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:09.377573 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61a2259-b0b0-453b-8b22-07c062919031 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:09.377857 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61a2259-b0b0-453b-8b22-07c062919031 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:09.378134 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61a2259-b0b0-453b-8b22-07c062919031 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:09.378781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61a2259-b0b0-453b-8b22-07c062919031 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:09.380771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61a2259-b0b0-453b-8b22-07c062919031 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:09.380947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-657b8991-7696-4903-956e-b7146eadca40 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:09.431765 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 744/1470] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:09 2022] POST /identity/v3/users => generated 300 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:09.434501 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61a2259-b0b0-453b-8b22-07c062919031 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:09.457276 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61a2259-b0b0-453b-8b22-07c062919031 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:09.467515 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298a52af-7f24-4241-a16e-21e2df187e5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:09.468196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298a52af-7f24-4241-a16e-21e2df187e5a admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:09.469249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298a52af-7f24-4241-a16e-21e2df187e5a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:09.469589 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298a52af-7f24-4241-a16e-21e2df187e5a admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:09.470545 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298a52af-7f24-4241-a16e-21e2df187e5a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:09.474576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298a52af-7f24-4241-a16e-21e2df187e5a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:09.492520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 745/1471] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:09 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:09.493936 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01df1133-fadc-4629-8cc5-0f10d67f7c5c None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:09.494222 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01df1133-fadc-4629-8cc5-0f10d67f7c5c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:09.494495 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01df1133-fadc-4629-8cc5-0f10d67f7c5c None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:09.496492 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 746/1472] 10.222.0.36 () {54 vars in 921 bytes} [Mon Oct 31 14:57:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:57:09.497527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01df1133-fadc-4629-8cc5-0f10d67f7c5c None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:09.511309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01df1133-fadc-4629-8cc5-0f10d67f7c5c None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:09.524295 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3124d77f-1ee0-4c94-815e-4d39aabdf2cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:09.524957 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3124d77f-1ee0-4c94-815e-4d39aabdf2cd admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:09.525325 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3124d77f-1ee0-4c94-815e-4d39aabdf2cd admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:09.525718 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3124d77f-1ee0-4c94-815e-4d39aabdf2cd admin admin] PATH_INFO: `/v3/projects/817fca98a8c347ecbbb305ba723a27d5/users/50a71eb4b28040b2973686447755d0c0/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:09.553476 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657b8991-7696-4903-956e-b7146eadca40 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:09.555323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657b8991-7696-4903-956e-b7146eadca40 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:09.559793 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3124d77f-1ee0-4c94-815e-4d39aabdf2cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=817fca98a8c347ecbbb305ba723a27d5, user_id=50a71eb4b28040b2973686447755d0c0, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:09.565067 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3124d77f-1ee0-4c94-815e-4d39aabdf2cd admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:09.585011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 727/1473] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:09 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 514 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:09.591382 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 747/1474] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:09 2022] PUT /identity/v3/projects/817fca98a8c347ecbbb305ba723a27d5/users/50a71eb4b28040b2973686447755d0c0/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:09.595447 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657b8991-7696-4903-956e-b7146eadca40 service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:09.616573 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657b8991-7696-4903-956e-b7146eadca40 service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:09.629957 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94b98fa-344b-4b73-8f05-d1cd83af6b08 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:09.630357 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94b98fa-344b-4b73-8f05-d1cd83af6b08 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:09.630522 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94b98fa-344b-4b73-8f05-d1cd83af6b08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:09.644002 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:09.675717 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-a94b98fa-344b-4b73-8f05-d1cd83af6b08 None None] MFA Rules not processed for user `c8cace029a92406f850bb0a4ca07c461`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:09.676987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a94b98fa-344b-4b73-8f05-d1cd83af6b08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:09.765937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7215a6b-7f42-438b-b4f2-7afa91162fc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:09.766496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7215a6b-7f42-438b-b4f2-7afa91162fc2 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:09.766783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7215a6b-7f42-438b-b4f2-7afa91162fc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:09.767062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7215a6b-7f42-438b-b4f2-7afa91162fc2 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:09.773883 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7215a6b-7f42-438b-b4f2-7afa91162fc2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:09.779475 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7215a6b-7f42-438b-b4f2-7afa91162fc2 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:09.821792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 728/1475] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:09 2022] GET /identity/v3/roles => generated 1420 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:09.827129 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7215a6b-7f42-438b-b4f2-7afa91162fc2 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:09.831468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 748/1476] 10.222.0.36 () {58 vars in 1018 bytes} [Mon Oct 31 14:57:09 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:57:09.870004 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7215a6b-7f42-438b-b4f2-7afa91162fc2 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:09.892414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1f139e-6b89-42ac-ac6e-4b8a23dbff9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:09.893147 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1f139e-6b89-42ac-ac6e-4b8a23dbff9d admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:09.895168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1f139e-6b89-42ac-ac6e-4b8a23dbff9d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:09.895614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1f139e-6b89-42ac-ac6e-4b8a23dbff9d admin admin] PATH_INFO: `/v3/projects/817fca98a8c347ecbbb305ba723a27d5/users/50a71eb4b28040b2973686447755d0c0/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:09.906407 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1f139e-6b89-42ac-ac6e-4b8a23dbff9d admin admin] RBAC: Authorizing `identity:create_grant(project_id=817fca98a8c347ecbbb305ba723a27d5, user_id=50a71eb4b28040b2973686447755d0c0, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:09.920957 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1f139e-6b89-42ac-ac6e-4b8a23dbff9d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:09.967713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 729/1477] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:09 2022] PUT /identity/v3/projects/817fca98a8c347ecbbb305ba723a27d5/users/50a71eb4b28040b2973686447755d0c0/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:09.986652 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689131e5-0e4f-4e64-be9f-163607b13f6c None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:09.986939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689131e5-0e4f-4e64-be9f-163607b13f6c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:09.987215 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689131e5-0e4f-4e64-be9f-163607b13f6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:10.053251 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-689131e5-0e4f-4e64-be9f-163607b13f6c None None] MFA Rules not processed for user `50a71eb4b28040b2973686447755d0c0`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:10.055428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-689131e5-0e4f-4e64-be9f-163607b13f6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:10.237140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 749/1478] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:09 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:10.317231 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1f139e-6b89-42ac-ac6e-4b8a23dbff9d admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:10.339693 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:10.429612 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1f139e-6b89-42ac-ac6e-4b8a23dbff9d admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:10.435521 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507e56bb-5722-4fa4-a234-49601838e020 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:10.436832 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507e56bb-5722-4fa4-a234-49601838e020 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:10.437257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507e56bb-5722-4fa4-a234-49601838e020 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:10.437607 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507e56bb-5722-4fa4-a234-49601838e020 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:10.456033 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507e56bb-5722-4fa4-a234-49601838e020 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:10.457675 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507e56bb-5722-4fa4-a234-49601838e020 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:10.493920 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 730/1479] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:10 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:10.608600 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689131e5-0e4f-4e64-be9f-163607b13f6c None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:10.681577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507e56bb-5722-4fa4-a234-49601838e020 service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:10.683643 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-507e56bb-5722-4fa4-a234-49601838e020 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:10.731968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689131e5-0e4f-4e64-be9f-163607b13f6c None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:10.737427 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a70d0ba-a786-428f-aa2b-8cb9b20a40ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:10.738488 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a70d0ba-a786-428f-aa2b-8cb9b20a40ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:10.738992 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a70d0ba-a786-428f-aa2b-8cb9b20a40ab admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:10.739469 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a70d0ba-a786-428f-aa2b-8cb9b20a40ab admin admin] PATH_INFO: `/v3/users/b00c33424f954948af34cae0907f786c` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:10.742625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a70d0ba-a786-428f-aa2b-8cb9b20a40ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=b00c33424f954948af34cae0907f786c)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:10.744844 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a70d0ba-a786-428f-aa2b-8cb9b20a40ab admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:10.779318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:10.811834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0a70d0ba-a786-428f-aa2b-8cb9b20a40ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b00c33424f954948af34cae0907f786c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:10.812501 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0a70d0ba-a786-428f-aa2b-8cb9b20a40ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b00c33424f954948af34cae0907f786c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:10.827784 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507e56bb-5722-4fa4-a234-49601838e020 service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:10.828037 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0a70d0ba-a786-428f-aa2b-8cb9b20a40ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b00c33424f954948af34cae0907f786c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:10.836359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d506d3f4-943e-4b9d-a866-1a16ebc5d209 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:10.837098 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d506d3f4-943e-4b9d-a866-1a16ebc5d209 service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:10.837512 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d506d3f4-943e-4b9d-a866-1a16ebc5d209 service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:10.837849 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d506d3f4-943e-4b9d-a866-1a16ebc5d209 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:10.840313 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d506d3f4-943e-4b9d-a866-1a16ebc5d209 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:10.841049 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0a70d0ba-a786-428f-aa2b-8cb9b20a40ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b00c33424f954948af34cae0907f786c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:10.851198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0a70d0ba-a786-428f-aa2b-8cb9b20a40ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b00c33424f954948af34cae0907f786c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:10.854663 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d506d3f4-943e-4b9d-a866-1a16ebc5d209 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:10.856764 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d506d3f4-943e-4b9d-a866-1a16ebc5d209 service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:10.864160 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 750/1480] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:10 2022] DELETE /identity/v3/users/b00c33424f954948af34cae0907f786c => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:10.866862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25ad761-c623-4593-b07d-e701db7cbefe None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:10.867969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25ad761-c623-4593-b07d-e701db7cbefe None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:10.868417 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25ad761-c623-4593-b07d-e701db7cbefe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:10.872513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 731/1481] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:10 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:10.891364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-e25ad761-c623-4593-b07d-e701db7cbefe None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:10.892799 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e25ad761-c623-4593-b07d-e701db7cbefe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:10.971268 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 751/1482] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:10.984766 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d506d3f4-943e-4b9d-a866-1a16ebc5d209 service nova] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:10.987386 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d506d3f4-943e-4b9d-a866-1a16ebc5d209 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:11.054365 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d506d3f4-943e-4b9d-a866-1a16ebc5d209 service nova] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:11.160060 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e355f662-7c54-4360-b7dd-486fb2c65dbc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:11.160216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e355f662-7c54-4360-b7dd-486fb2c65dbc service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:11.160425 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e355f662-7c54-4360-b7dd-486fb2c65dbc service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:11.160993 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e355f662-7c54-4360-b7dd-486fb2c65dbc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:11.173090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e355f662-7c54-4360-b7dd-486fb2c65dbc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:11.174554 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e355f662-7c54-4360-b7dd-486fb2c65dbc service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:11.182109 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:11.190327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 732/1483] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:11.552790 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25ad761-c623-4593-b07d-e701db7cbefe None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:11.554325 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e25ad761-c623-4593-b07d-e701db7cbefe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:11.573422 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25ad761-c623-4593-b07d-e701db7cbefe None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:11.578527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a716e397-a86a-4dc2-b5aa-be5760e6895e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:11.579159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a716e397-a86a-4dc2-b5aa-be5760e6895e admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:11.579437 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a716e397-a86a-4dc2-b5aa-be5760e6895e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:11.579742 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a716e397-a86a-4dc2-b5aa-be5760e6895e admin admin] PATH_INFO: `/v3/projects/966e35ad3ebe4c969cc5bba3bef68c57` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:11.580850 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a716e397-a86a-4dc2-b5aa-be5760e6895e admin admin] RBAC: Authorizing `identity:delete_project(project_id=966e35ad3ebe4c969cc5bba3bef68c57)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:11.582823 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a716e397-a86a-4dc2-b5aa-be5760e6895e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:11.640836 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a716e397-a86a-4dc2-b5aa-be5760e6895e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '966e35ad3ebe4c969cc5bba3bef68c57', 'request_id': 'req-a716e397-a86a-4dc2-b5aa-be5760e6895e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:11.647072 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a716e397-a86a-4dc2-b5aa-be5760e6895e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '966e35ad3ebe4c969cc5bba3bef68c57', 'request_id': 'req-a716e397-a86a-4dc2-b5aa-be5760e6895e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:11.654971 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a716e397-a86a-4dc2-b5aa-be5760e6895e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '966e35ad3ebe4c969cc5bba3bef68c57', 'request_id': 'req-a716e397-a86a-4dc2-b5aa-be5760e6895e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:11.670033 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a716e397-a86a-4dc2-b5aa-be5760e6895e admin admin] The token cache is being invalidate because project 966e35ad3ebe4c969cc5bba3bef68c57 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:11.670350 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a716e397-a86a-4dc2-b5aa-be5760e6895e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:11.671088 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a716e397-a86a-4dc2-b5aa-be5760e6895e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:11.672650 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 752/1484] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:11 2022] DELETE /identity/v3/projects/966e35ad3ebe4c969cc5bba3bef68c57 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:11.676901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e355f662-7c54-4360-b7dd-486fb2c65dbc service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:11.689431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e355f662-7c54-4360-b7dd-486fb2c65dbc service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:11.761471 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47278b8b-46a0-4875-85f5-65a346681d48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:11.762241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47278b8b-46a0-4875-85f5-65a346681d48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:11.762521 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47278b8b-46a0-4875-85f5-65a346681d48 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:11.762792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47278b8b-46a0-4875-85f5-65a346681d48 admin admin] PATH_INFO: `/v3/users/50a71eb4b28040b2973686447755d0c0` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:11.764590 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47278b8b-46a0-4875-85f5-65a346681d48 admin admin] RBAC: Authorizing `identity:delete_user(user_id=50a71eb4b28040b2973686447755d0c0)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:11.766775 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47278b8b-46a0-4875-85f5-65a346681d48 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:11.832365 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-47278b8b-46a0-4875-85f5-65a346681d48 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '50a71eb4b28040b2973686447755d0c0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:11.844288 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-47278b8b-46a0-4875-85f5-65a346681d48 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '50a71eb4b28040b2973686447755d0c0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:11.845158 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-47278b8b-46a0-4875-85f5-65a346681d48 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '50a71eb4b28040b2973686447755d0c0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:11.845906 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-47278b8b-46a0-4875-85f5-65a346681d48 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '50a71eb4b28040b2973686447755d0c0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:11.860233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-47278b8b-46a0-4875-85f5-65a346681d48 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '50a71eb4b28040b2973686447755d0c0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:11.874508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 733/1485] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:11 2022] DELETE /identity/v3/users/50a71eb4b28040b2973686447755d0c0 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:12.192723 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a716e397-a86a-4dc2-b5aa-be5760e6895e admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:12.195233 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a716e397-a86a-4dc2-b5aa-be5760e6895e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:12.210750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a716e397-a86a-4dc2-b5aa-be5760e6895e admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:12.239916 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:12.284910 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7318e2-fd1e-40d8-87ff-958d57a8d0e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:12.285558 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7318e2-fd1e-40d8-87ff-958d57a8d0e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:12.285840 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7318e2-fd1e-40d8-87ff-958d57a8d0e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:12.286113 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7318e2-fd1e-40d8-87ff-958d57a8d0e7 admin admin] PATH_INFO: `/v3/projects/817fca98a8c347ecbbb305ba723a27d5` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:12.287227 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7318e2-fd1e-40d8-87ff-958d57a8d0e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=817fca98a8c347ecbbb305ba723a27d5)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:12.289167 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7318e2-fd1e-40d8-87ff-958d57a8d0e7 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:12.335993 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-fc7318e2-fd1e-40d8-87ff-958d57a8d0e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '817fca98a8c347ecbbb305ba723a27d5', 'request_id': 'req-fc7318e2-fd1e-40d8-87ff-958d57a8d0e7'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:12.341507 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-fc7318e2-fd1e-40d8-87ff-958d57a8d0e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '817fca98a8c347ecbbb305ba723a27d5', 'request_id': 'req-fc7318e2-fd1e-40d8-87ff-958d57a8d0e7'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:12.347589 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-fc7318e2-fd1e-40d8-87ff-958d57a8d0e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '817fca98a8c347ecbbb305ba723a27d5', 'request_id': 'req-fc7318e2-fd1e-40d8-87ff-958d57a8d0e7'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:12.361685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-fc7318e2-fd1e-40d8-87ff-958d57a8d0e7 admin admin] The token cache is being invalidate because project 817fca98a8c347ecbbb305ba723a27d5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:12.361973 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-fc7318e2-fd1e-40d8-87ff-958d57a8d0e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:12.362607 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-fc7318e2-fd1e-40d8-87ff-958d57a8d0e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:12.364151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 753/1486] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:12 2022] DELETE /identity/v3/projects/817fca98a8c347ecbbb305ba723a27d5 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:12.369158 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a851324-d5ef-4d2b-9f52-8e8c8f76b5be None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:12.369780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a851324-d5ef-4d2b-9f52-8e8c8f76b5be None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:12.370124 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a851324-d5ef-4d2b-9f52-8e8c8f76b5be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:12.393142 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-1a851324-d5ef-4d2b-9f52-8e8c8f76b5be None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:12.394639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-1a851324-d5ef-4d2b-9f52-8e8c8f76b5be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:12.468202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 734/1487] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:12.477140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157d059f-cbc8-4ee5-af1f-1b6dc47fb1d5 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:12.477441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157d059f-cbc8-4ee5-af1f-1b6dc47fb1d5 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:12.477765 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157d059f-cbc8-4ee5-af1f-1b6dc47fb1d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:12.499801 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-157d059f-cbc8-4ee5-af1f-1b6dc47fb1d5 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:12.500962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-157d059f-cbc8-4ee5-af1f-1b6dc47fb1d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:12.509138 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 754/1488] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:12.513565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a851324-d5ef-4d2b-9f52-8e8c8f76b5be None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:12.523354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a851324-d5ef-4d2b-9f52-8e8c8f76b5be None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:12.528451 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c1b5c4-9acb-4224-9b1b-32225b6f2700 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:12.529060 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c1b5c4-9acb-4224-9b1b-32225b6f2700 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:12.529337 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c1b5c4-9acb-4224-9b1b-32225b6f2700 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:12.529613 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c1b5c4-9acb-4224-9b1b-32225b6f2700 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:12.530286 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c1b5c4-9acb-4224-9b1b-32225b6f2700 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:57:12.530569 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c1b5c4-9acb-4224-9b1b-32225b6f2700 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:12.532215 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c1b5c4-9acb-4224-9b1b-32225b6f2700 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:12.544222 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 735/1489] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:57:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:57:12.548518 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157d059f-cbc8-4ee5-af1f-1b6dc47fb1d5 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:12.557632 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157d059f-cbc8-4ee5-af1f-1b6dc47fb1d5 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:12.563320 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae33125-1f59-4987-b78e-f6ef97d1b491 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:12.563946 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae33125-1f59-4987-b78e-f6ef97d1b491 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:12.564226 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae33125-1f59-4987-b78e-f6ef97d1b491 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:12.564498 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae33125-1f59-4987-b78e-f6ef97d1b491 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:12.565139 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae33125-1f59-4987-b78e-f6ef97d1b491 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:12.567022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae33125-1f59-4987-b78e-f6ef97d1b491 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:12.594311 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 755/1490] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:12 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:12.598437 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c1b5c4-9acb-4224-9b1b-32225b6f2700 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:12.608786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c1b5c4-9acb-4224-9b1b-32225b6f2700 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:12.612247 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:12.666614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5479f5e3-e566-4ce4-8a11-744aec2e45de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:12.667213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5479f5e3-e566-4ce4-8a11-744aec2e45de admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:12.668153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5479f5e3-e566-4ce4-8a11-744aec2e45de admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:12.668524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5479f5e3-e566-4ce4-8a11-744aec2e45de admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:12.669159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5479f5e3-e566-4ce4-8a11-744aec2e45de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:12.671059 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5479f5e3-e566-4ce4-8a11-744aec2e45de admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:12.694153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 736/1491] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:12 2022] POST /identity/v3/users => generated 288 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:12.695357 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae33125-1f59-4987-b78e-f6ef97d1b491 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:12.705704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae33125-1f59-4987-b78e-f6ef97d1b491 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:12.710549 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1839be84-0f39-4970-b839-36cae6ae81df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:12.711142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1839be84-0f39-4970-b839-36cae6ae81df admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:12.711421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1839be84-0f39-4970-b839-36cae6ae81df admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:12.711732 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1839be84-0f39-4970-b839-36cae6ae81df admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:12.712520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1839be84-0f39-4970-b839-36cae6ae81df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:12.714094 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1839be84-0f39-4970-b839-36cae6ae81df admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:12.724212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 756/1492] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:12 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:12.728074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5479f5e3-e566-4ce4-8a11-744aec2e45de admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:12.738806 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5479f5e3-e566-4ce4-8a11-744aec2e45de admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:12.743507 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c071f25f-b3da-49e3-ada8-34483b4b5929 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:12.744133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c071f25f-b3da-49e3-ada8-34483b4b5929 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:12.744412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c071f25f-b3da-49e3-ada8-34483b4b5929 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:12.744710 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c071f25f-b3da-49e3-ada8-34483b4b5929 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:12.745351 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c071f25f-b3da-49e3-ada8-34483b4b5929 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:12.746970 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c071f25f-b3da-49e3-ada8-34483b4b5929 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:12.757556 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 737/1493] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:12.761734 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1839be84-0f39-4970-b839-36cae6ae81df admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:12.771700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1839be84-0f39-4970-b839-36cae6ae81df admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:12.776377 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebef986e-4a59-4c04-9c17-4da7fbbd852f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:12.777498 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebef986e-4a59-4c04-9c17-4da7fbbd852f admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:12.777891 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebef986e-4a59-4c04-9c17-4da7fbbd852f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:12.778205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebef986e-4a59-4c04-9c17-4da7fbbd852f admin admin] PATH_INFO: `/v3/projects/7308e9a0e8b24dd68e7031770f9bd368/users/b0a4f3bff51b4304bcfe4434db9c0f76/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:12.792344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebef986e-4a59-4c04-9c17-4da7fbbd852f admin admin] RBAC: Authorizing `identity:create_grant(project_id=7308e9a0e8b24dd68e7031770f9bd368, user_id=b0a4f3bff51b4304bcfe4434db9c0f76, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:12.795597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebef986e-4a59-4c04-9c17-4da7fbbd852f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:12.808478 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 757/1494] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:12 2022] PUT /identity/v3/projects/7308e9a0e8b24dd68e7031770f9bd368/users/b0a4f3bff51b4304bcfe4434db9c0f76/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:12.818813 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369bb999-df1a-4fd8-b962-20615b31e4f8 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:12.819098 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369bb999-df1a-4fd8-b962-20615b31e4f8 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:12.819369 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369bb999-df1a-4fd8-b962-20615b31e4f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:12.848155 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-369bb999-df1a-4fd8-b962-20615b31e4f8 None None] MFA Rules not processed for user `b0a4f3bff51b4304bcfe4434db9c0f76`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:12.849430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-369bb999-df1a-4fd8-b962-20615b31e4f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:12.914119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 738/1495] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:12 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:12.926739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebef986e-4a59-4c04-9c17-4da7fbbd852f admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:12.928424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ebef986e-4a59-4c04-9c17-4da7fbbd852f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:13.000641 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebef986e-4a59-4c04-9c17-4da7fbbd852f admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:13.004361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55577b9b-403d-4507-8fa4-18daa33c1bb7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:13.004971 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55577b9b-403d-4507-8fa4-18daa33c1bb7 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:13.005247 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55577b9b-403d-4507-8fa4-18daa33c1bb7 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:13.005515 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55577b9b-403d-4507-8fa4-18daa33c1bb7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:13.017270 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:13.021915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55577b9b-403d-4507-8fa4-18daa33c1bb7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:13.024063 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55577b9b-403d-4507-8fa4-18daa33c1bb7 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:13.037651 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 758/1496] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:12 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:19.231463 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc65c93d-b964-4552-b948-9c3d5ea61890 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:19.231463 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc65c93d-b964-4552-b948-9c3d5ea61890 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:19.231463 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc65c93d-b964-4552-b948-9c3d5ea61890 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:19.231463 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 739/1497] 10.222.0.36 () {54 vars in 921 bytes} [Mon Oct 31 14:57:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:57:19.240454 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8727680-29a7-4448-b861-e77e05c2a3de None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:19.240849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8727680-29a7-4448-b861-e77e05c2a3de None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:19.241144 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8727680-29a7-4448-b861-e77e05c2a3de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:19.262088 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-d8727680-29a7-4448-b861-e77e05c2a3de None None] MFA Rules not processed for user `c8cace029a92406f850bb0a4ca07c461`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:19.263318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d8727680-29a7-4448-b861-e77e05c2a3de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:19.333067 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 759/1498] 10.222.0.36 () {58 vars in 1018 bytes} [Mon Oct 31 14:57:19 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:19.345779 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc65c93d-b964-4552-b948-9c3d5ea61890 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:19.347475 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-dc65c93d-b964-4552-b948-9c3d5ea61890 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:19.365185 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc65c93d-b964-4552-b948-9c3d5ea61890 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:19.370684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcf0952-0d41-40c9-a2e8-8d9ce3c1236f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:19.371331 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcf0952-0d41-40c9-a2e8-8d9ce3c1236f service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:19.371651 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcf0952-0d41-40c9-a2e8-8d9ce3c1236f service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:19.371926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcf0952-0d41-40c9-a2e8-8d9ce3c1236f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:19.380446 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcf0952-0d41-40c9-a2e8-8d9ce3c1236f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:19.382035 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcf0952-0d41-40c9-a2e8-8d9ce3c1236f service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:19.392121 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 740/1499] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:21.071613 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8727680-29a7-4448-b861-e77e05c2a3de None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:21.084863 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8727680-29a7-4448-b861-e77e05c2a3de None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:21.088824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e1c960-f3b3-4deb-970a-30d5147a9746 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:21.089873 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e1c960-f3b3-4deb-970a-30d5147a9746 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:21.090322 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e1c960-f3b3-4deb-970a-30d5147a9746 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:21.090807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e1c960-f3b3-4deb-970a-30d5147a9746 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:21.093377 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-79e1c960-f3b3-4deb-970a-30d5147a9746 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:21.185259 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e1c960-f3b3-4deb-970a-30d5147a9746 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:21.186767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e1c960-f3b3-4deb-970a-30d5147a9746 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:21.196891 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 760/1500] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:21 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:24.694841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcf0952-0d41-40c9-a2e8-8d9ce3c1236f service nova] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:24.696316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-cdcf0952-0d41-40c9-a2e8-8d9ce3c1236f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:24.793747 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcf0952-0d41-40c9-a2e8-8d9ce3c1236f service nova] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:24.799686 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a1a15d-7984-40ac-922b-3c25b75f5e38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:24.800498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a1a15d-7984-40ac-922b-3c25b75f5e38 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:24.800926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a1a15d-7984-40ac-922b-3c25b75f5e38 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:24.801460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a1a15d-7984-40ac-922b-3c25b75f5e38 admin admin] PATH_INFO: `/v3/users/9d564924dc2347fc9aff513353941d84` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:24.802776 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a1a15d-7984-40ac-922b-3c25b75f5e38 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9d564924dc2347fc9aff513353941d84)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:24.815868 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a1a15d-7984-40ac-922b-3c25b75f5e38 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:24.819954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:24.885770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-85a1a15d-7984-40ac-922b-3c25b75f5e38 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9d564924dc2347fc9aff513353941d84'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:24.897200 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-85a1a15d-7984-40ac-922b-3c25b75f5e38 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9d564924dc2347fc9aff513353941d84'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:24.898258 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-85a1a15d-7984-40ac-922b-3c25b75f5e38 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9d564924dc2347fc9aff513353941d84'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:24.899258 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-85a1a15d-7984-40ac-922b-3c25b75f5e38 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9d564924dc2347fc9aff513353941d84'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:24.916465 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-85a1a15d-7984-40ac-922b-3c25b75f5e38 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9d564924dc2347fc9aff513353941d84'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:24.941818 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 741/1501] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:24 2022] DELETE /identity/v3/users/9d564924dc2347fc9aff513353941d84 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:24.951690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5ee7af-86b4-4890-b5b2-480135a19afe None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:24.951690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5ee7af-86b4-4890-b5b2-480135a19afe None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:24.951690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5ee7af-86b4-4890-b5b2-480135a19afe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:24.976677 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-ac5ee7af-86b4-4890-b5b2-480135a19afe None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:24.978533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ac5ee7af-86b4-4890-b5b2-480135a19afe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:25.054489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 761/1502] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:25.069102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a1a15d-7984-40ac-922b-3c25b75f5e38 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:25.071027 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-85a1a15d-7984-40ac-922b-3c25b75f5e38 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:25.152850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a1a15d-7984-40ac-922b-3c25b75f5e38 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:25.158047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f54205-5877-4600-9342-19d02e5cc6ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:25.158752 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f54205-5877-4600-9342-19d02e5cc6ad service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:25.159042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f54205-5877-4600-9342-19d02e5cc6ad service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:25.159361 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f54205-5877-4600-9342-19d02e5cc6ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:25.173873 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f54205-5877-4600-9342-19d02e5cc6ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:25.175597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f54205-5877-4600-9342-19d02e5cc6ad service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:25.188711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 742/1503] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:25.704243 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5ee7af-86b4-4890-b5b2-480135a19afe None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:25.705710 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ac5ee7af-86b4-4890-b5b2-480135a19afe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:25.719465 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5ee7af-86b4-4890-b5b2-480135a19afe None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:25.724886 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8382452-5b7b-4593-a729-b51850821461 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:25.725724 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8382452-5b7b-4593-a729-b51850821461 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:25.726160 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8382452-5b7b-4593-a729-b51850821461 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:25.726575 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8382452-5b7b-4593-a729-b51850821461 admin admin] PATH_INFO: `/v3/projects/fbcaed4971c04c66b46ee6b90d6b54be` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:25.728071 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8382452-5b7b-4593-a729-b51850821461 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fbcaed4971c04c66b46ee6b90d6b54be)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:25.730397 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8382452-5b7b-4593-a729-b51850821461 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:25.753810 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:25.785682 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b8382452-5b7b-4593-a729-b51850821461 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fbcaed4971c04c66b46ee6b90d6b54be', 'request_id': 'req-b8382452-5b7b-4593-a729-b51850821461'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:25.805359 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b8382452-5b7b-4593-a729-b51850821461 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fbcaed4971c04c66b46ee6b90d6b54be', 'request_id': 'req-b8382452-5b7b-4593-a729-b51850821461'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:25.821184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b8382452-5b7b-4593-a729-b51850821461 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fbcaed4971c04c66b46ee6b90d6b54be', 'request_id': 'req-b8382452-5b7b-4593-a729-b51850821461'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:25.837981 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b8382452-5b7b-4593-a729-b51850821461 admin admin] The token cache is being invalidate because project fbcaed4971c04c66b46ee6b90d6b54be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:25.838274 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b8382452-5b7b-4593-a729-b51850821461 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:25.839455 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b8382452-5b7b-4593-a729-b51850821461 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:25.841156 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 762/1504] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:25 2022] DELETE /identity/v3/projects/fbcaed4971c04c66b46ee6b90d6b54be => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:25.845379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f54205-5877-4600-9342-19d02e5cc6ad service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:25.855419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f54205-5877-4600-9342-19d02e5cc6ad service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:25.917713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a6b35b-eaf0-4f67-a568-893043356712 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:25.918354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a6b35b-eaf0-4f67-a568-893043356712 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:25.918638 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a6b35b-eaf0-4f67-a568-893043356712 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:25.918916 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a6b35b-eaf0-4f67-a568-893043356712 admin admin] PATH_INFO: `/v3/users/c6cabcb2eb45410199ed41806ffa9958` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:25.920093 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a6b35b-eaf0-4f67-a568-893043356712 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c6cabcb2eb45410199ed41806ffa9958)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:25.922037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a6b35b-eaf0-4f67-a568-893043356712 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:25.953789 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:25.976204 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-78a6b35b-eaf0-4f67-a568-893043356712 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c6cabcb2eb45410199ed41806ffa9958'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:25.983439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-78a6b35b-eaf0-4f67-a568-893043356712 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c6cabcb2eb45410199ed41806ffa9958'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:25.984145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-78a6b35b-eaf0-4f67-a568-893043356712 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c6cabcb2eb45410199ed41806ffa9958'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:25.984758 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-78a6b35b-eaf0-4f67-a568-893043356712 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c6cabcb2eb45410199ed41806ffa9958'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:25.997920 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-78a6b35b-eaf0-4f67-a568-893043356712 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c6cabcb2eb45410199ed41806ffa9958'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:26.012577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 743/1505] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:25 2022] DELETE /identity/v3/users/c6cabcb2eb45410199ed41806ffa9958 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:26.371176 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8382452-5b7b-4593-a729-b51850821461 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:26.372647 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b8382452-5b7b-4593-a729-b51850821461 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:26.385233 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8382452-5b7b-4593-a729-b51850821461 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:26.443127 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce85883-07cc-49b9-96f6-c976f1794694 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:26.443790 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce85883-07cc-49b9-96f6-c976f1794694 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:26.444068 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce85883-07cc-49b9-96f6-c976f1794694 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:26.444335 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce85883-07cc-49b9-96f6-c976f1794694 admin admin] PATH_INFO: `/v3/projects/78d7dc6835e64bc19dc66cb00c3f47ee` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:26.445433 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce85883-07cc-49b9-96f6-c976f1794694 admin admin] RBAC: Authorizing `identity:delete_project(project_id=78d7dc6835e64bc19dc66cb00c3f47ee)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:26.447382 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce85883-07cc-49b9-96f6-c976f1794694 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:26.494005 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-dce85883-07cc-49b9-96f6-c976f1794694 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '78d7dc6835e64bc19dc66cb00c3f47ee', 'request_id': 'req-dce85883-07cc-49b9-96f6-c976f1794694'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:26.499994 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-dce85883-07cc-49b9-96f6-c976f1794694 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '78d7dc6835e64bc19dc66cb00c3f47ee', 'request_id': 'req-dce85883-07cc-49b9-96f6-c976f1794694'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:26.506035 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-dce85883-07cc-49b9-96f6-c976f1794694 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '78d7dc6835e64bc19dc66cb00c3f47ee', 'request_id': 'req-dce85883-07cc-49b9-96f6-c976f1794694'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:26.518708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-dce85883-07cc-49b9-96f6-c976f1794694 admin admin] The token cache is being invalidate because project 78d7dc6835e64bc19dc66cb00c3f47ee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:26.518996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-dce85883-07cc-49b9-96f6-c976f1794694 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:26.519755 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-dce85883-07cc-49b9-96f6-c976f1794694 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:26.523637 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 763/1506] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:26 2022] DELETE /identity/v3/projects/78d7dc6835e64bc19dc66cb00c3f47ee => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:26.525329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a6b35b-eaf0-4f67-a568-893043356712 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:26.526761 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-78a6b35b-eaf0-4f67-a568-893043356712 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:26.539972 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a6b35b-eaf0-4f67-a568-893043356712 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:26.599198 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8649b039-a8b3-41ca-8166-1dd07aceddb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:26.599874 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8649b039-a8b3-41ca-8166-1dd07aceddb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:26.600158 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8649b039-a8b3-41ca-8166-1dd07aceddb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:26.600435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8649b039-a8b3-41ca-8166-1dd07aceddb8 admin admin] PATH_INFO: `/v3/users/4c68892fdcb34d0cbec882eba28b097a` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:26.601551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8649b039-a8b3-41ca-8166-1dd07aceddb8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4c68892fdcb34d0cbec882eba28b097a)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:26.603474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8649b039-a8b3-41ca-8166-1dd07aceddb8 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:26.651150 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8649b039-a8b3-41ca-8166-1dd07aceddb8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4c68892fdcb34d0cbec882eba28b097a'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:26.658064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8649b039-a8b3-41ca-8166-1dd07aceddb8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4c68892fdcb34d0cbec882eba28b097a'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:26.658728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8649b039-a8b3-41ca-8166-1dd07aceddb8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4c68892fdcb34d0cbec882eba28b097a'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:26.659333 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8649b039-a8b3-41ca-8166-1dd07aceddb8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4c68892fdcb34d0cbec882eba28b097a'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:26.671971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8649b039-a8b3-41ca-8166-1dd07aceddb8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4c68892fdcb34d0cbec882eba28b097a'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:26.684795 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 744/1507] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:26 2022] DELETE /identity/v3/users/4c68892fdcb34d0cbec882eba28b097a => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:27.110211 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce85883-07cc-49b9-96f6-c976f1794694 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:27.112187 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-dce85883-07cc-49b9-96f6-c976f1794694 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:27.125688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce85883-07cc-49b9-96f6-c976f1794694 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:27.162676 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:27.188854 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ecf95e-16b1-492f-a63e-31b143056e34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:27.189496 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ecf95e-16b1-492f-a63e-31b143056e34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:27.189776 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ecf95e-16b1-492f-a63e-31b143056e34 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:27.190046 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ecf95e-16b1-492f-a63e-31b143056e34 admin admin] PATH_INFO: `/v3/projects/d21354c3e7ae481dbd38c5e3deead403` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:27.191196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ecf95e-16b1-492f-a63e-31b143056e34 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d21354c3e7ae481dbd38c5e3deead403)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:27.193138 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ecf95e-16b1-492f-a63e-31b143056e34 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:27.239504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-94ecf95e-16b1-492f-a63e-31b143056e34 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd21354c3e7ae481dbd38c5e3deead403', 'request_id': 'req-94ecf95e-16b1-492f-a63e-31b143056e34'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:27.244977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-94ecf95e-16b1-492f-a63e-31b143056e34 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd21354c3e7ae481dbd38c5e3deead403', 'request_id': 'req-94ecf95e-16b1-492f-a63e-31b143056e34'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:27.251081 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-94ecf95e-16b1-492f-a63e-31b143056e34 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd21354c3e7ae481dbd38c5e3deead403', 'request_id': 'req-94ecf95e-16b1-492f-a63e-31b143056e34'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:27.263668 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-94ecf95e-16b1-492f-a63e-31b143056e34 admin admin] The token cache is being invalidate because project d21354c3e7ae481dbd38c5e3deead403 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:27.263964 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-94ecf95e-16b1-492f-a63e-31b143056e34 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:27.264590 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-94ecf95e-16b1-492f-a63e-31b143056e34 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:27.266083 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 764/1508] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:27 2022] DELETE /identity/v3/projects/d21354c3e7ae481dbd38c5e3deead403 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:27.272256 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcf525a-a8e8-4877-ae34-cb8191ae2d8b None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:27.272734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcf525a-a8e8-4877-ae34-cb8191ae2d8b None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:27.272823 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcf525a-a8e8-4877-ae34-cb8191ae2d8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:27.293658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-3dcf525a-a8e8-4877-ae34-cb8191ae2d8b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:27.294858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-3dcf525a-a8e8-4877-ae34-cb8191ae2d8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:27.312560 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:DELETE=16,keystone:SELECT=82,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:27.356398 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 745/1509] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:27.366013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d34471-4dc6-49ce-b713-add08dcc27c6 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:27.366303 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d34471-4dc6-49ce-b713-add08dcc27c6 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:27.366591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d34471-4dc6-49ce-b713-add08dcc27c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:27.391139 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-19d34471-4dc6-49ce-b713-add08dcc27c6 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:27.392381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-19d34471-4dc6-49ce-b713-add08dcc27c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:27.404120 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 765/1510] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:27.406284 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcf525a-a8e8-4877-ae34-cb8191ae2d8b None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:27.417042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcf525a-a8e8-4877-ae34-cb8191ae2d8b None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:27.421955 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d677cc6f-bad3-4e1a-a23d-db79c24d6cb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:27.422556 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d677cc6f-bad3-4e1a-a23d-db79c24d6cb9 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:27.422835 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d677cc6f-bad3-4e1a-a23d-db79c24d6cb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:27.423108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d677cc6f-bad3-4e1a-a23d-db79c24d6cb9 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:27.423929 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d677cc6f-bad3-4e1a-a23d-db79c24d6cb9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:57:27.424233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d677cc6f-bad3-4e1a-a23d-db79c24d6cb9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:27.425982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d677cc6f-bad3-4e1a-a23d-db79c24d6cb9 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:27.439079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 746/1511] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:57:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:57:27.443087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d34471-4dc6-49ce-b713-add08dcc27c6 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:27.452829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d34471-4dc6-49ce-b713-add08dcc27c6 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:27.457521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec361de-6f41-44cd-b8a8-f6cf404d2c22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:27.458384 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec361de-6f41-44cd-b8a8-f6cf404d2c22 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:27.458666 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec361de-6f41-44cd-b8a8-f6cf404d2c22 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:27.458941 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec361de-6f41-44cd-b8a8-f6cf404d2c22 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:27.459661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec361de-6f41-44cd-b8a8-f6cf404d2c22 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:27.461566 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec361de-6f41-44cd-b8a8-f6cf404d2c22 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:27.493762 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 766/1512] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:27 2022] POST /identity/v3/projects => generated 407 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:27.495495 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d677cc6f-bad3-4e1a-a23d-db79c24d6cb9 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:27.505767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d677cc6f-bad3-4e1a-a23d-db79c24d6cb9 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:27.568688 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1329a1-c1d5-4020-bcb3-46776d4101e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:27.569295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1329a1-c1d5-4020-bcb3-46776d4101e8 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:27.569580 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1329a1-c1d5-4020-bcb3-46776d4101e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:27.569855 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1329a1-c1d5-4020-bcb3-46776d4101e8 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:27.570492 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1329a1-c1d5-4020-bcb3-46776d4101e8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:27.572450 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1329a1-c1d5-4020-bcb3-46776d4101e8 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:27.595921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 747/1513] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:27 2022] POST /identity/v3/users => generated 304 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:57:27.600075 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec361de-6f41-44cd-b8a8-f6cf404d2c22 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:27.611162 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec361de-6f41-44cd-b8a8-f6cf404d2c22 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:27.616066 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac8f3e1-71d4-4490-b210-14a282af7efb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:27.616670 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac8f3e1-71d4-4490-b210-14a282af7efb admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:27.616947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac8f3e1-71d4-4490-b210-14a282af7efb admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:27.617249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac8f3e1-71d4-4490-b210-14a282af7efb admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:27.617893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac8f3e1-71d4-4490-b210-14a282af7efb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:27.619461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac8f3e1-71d4-4490-b210-14a282af7efb admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:27.630135 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 767/1514] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:27.634223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1329a1-c1d5-4020-bcb3-46776d4101e8 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:27.644967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1329a1-c1d5-4020-bcb3-46776d4101e8 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:27.649678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5068524-7227-4bbc-8b5f-0be7cafc9a13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:27.650285 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5068524-7227-4bbc-8b5f-0be7cafc9a13 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:27.650562 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5068524-7227-4bbc-8b5f-0be7cafc9a13 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:27.650911 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5068524-7227-4bbc-8b5f-0be7cafc9a13 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:27.651712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5068524-7227-4bbc-8b5f-0be7cafc9a13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:27.653317 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5068524-7227-4bbc-8b5f-0be7cafc9a13 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:27.664787 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 748/1515] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:27.669244 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac8f3e1-71d4-4490-b210-14a282af7efb admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:27.680213 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac8f3e1-71d4-4490-b210-14a282af7efb admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:27.686547 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4af606-5577-428a-a63f-d291a7a6e17f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:27.687318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4af606-5577-428a-a63f-d291a7a6e17f admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:27.687633 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4af606-5577-428a-a63f-d291a7a6e17f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:27.687910 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4af606-5577-428a-a63f-d291a7a6e17f admin admin] PATH_INFO: `/v3/projects/e9b4db8138434ee4ae5d9533c04f8b2a/users/0efa2d3ea582464398471fa1780c4370/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:27.703458 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4af606-5577-428a-a63f-d291a7a6e17f admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9b4db8138434ee4ae5d9533c04f8b2a, user_id=0efa2d3ea582464398471fa1780c4370, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:27.706126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4af606-5577-428a-a63f-d291a7a6e17f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:27.720374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 768/1516] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:27 2022] PUT /identity/v3/projects/e9b4db8138434ee4ae5d9533c04f8b2a/users/0efa2d3ea582464398471fa1780c4370/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:27.731823 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63e6e30-75ba-4e10-950c-1cc4fce33a15 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:27.732111 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63e6e30-75ba-4e10-950c-1cc4fce33a15 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:27.732387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63e6e30-75ba-4e10-950c-1cc4fce33a15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:27.764962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-a63e6e30-75ba-4e10-950c-1cc4fce33a15 None None] MFA Rules not processed for user `0efa2d3ea582464398471fa1780c4370`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:27.766276 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a63e6e30-75ba-4e10-950c-1cc4fce33a15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:27.832694 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 749/1517] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:27 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:27.837740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4af606-5577-428a-a63f-d291a7a6e17f admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:27.904886 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4af606-5577-428a-a63f-d291a7a6e17f admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:27.909746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0899b65d-1af4-439c-baaf-6f494f3f2794 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:27.910357 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0899b65d-1af4-439c-baaf-6f494f3f2794 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:27.910670 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0899b65d-1af4-439c-baaf-6f494f3f2794 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:27.910909 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0899b65d-1af4-439c-baaf-6f494f3f2794 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:27.911538 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0899b65d-1af4-439c-baaf-6f494f3f2794 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:27.913462 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0899b65d-1af4-439c-baaf-6f494f3f2794 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:27.924485 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:27.946975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 769/1518] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:27 2022] POST /identity/v3/projects => generated 407 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:27.950962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63e6e30-75ba-4e10-950c-1cc4fce33a15 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:27.962542 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63e6e30-75ba-4e10-950c-1cc4fce33a15 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:27.975329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:28.030208 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aff55e3-9cac-4f1b-b3ad-96847b773beb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:28.030734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aff55e3-9cac-4f1b-b3ad-96847b773beb admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:28.031026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aff55e3-9cac-4f1b-b3ad-96847b773beb admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:28.031300 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aff55e3-9cac-4f1b-b3ad-96847b773beb admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:28.031982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aff55e3-9cac-4f1b-b3ad-96847b773beb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:28.033968 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aff55e3-9cac-4f1b-b3ad-96847b773beb admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:28.072970 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 750/1519] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:27 2022] POST /identity/v3/users => generated 310 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:28.073704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0899b65d-1af4-439c-baaf-6f494f3f2794 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:28.084604 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0899b65d-1af4-439c-baaf-6f494f3f2794 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:28.089621 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b655e0df-7674-404d-a706-c91698022099 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:28.090244 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b655e0df-7674-404d-a706-c91698022099 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:28.090844 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b655e0df-7674-404d-a706-c91698022099 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:28.091091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b655e0df-7674-404d-a706-c91698022099 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:28.091774 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b655e0df-7674-404d-a706-c91698022099 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:28.093378 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b655e0df-7674-404d-a706-c91698022099 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:28.107607 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 770/1520] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:28 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:28.109792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aff55e3-9cac-4f1b-b3ad-96847b773beb admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:28.122218 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aff55e3-9cac-4f1b-b3ad-96847b773beb admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:28.127919 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44da61bd-c39c-487d-9ca9-d1202a264135 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:28.128687 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44da61bd-c39c-487d-9ca9-d1202a264135 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:28.129103 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44da61bd-c39c-487d-9ca9-d1202a264135 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:28.136208 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44da61bd-c39c-487d-9ca9-d1202a264135 admin admin] PATH_INFO: `/v3/projects/e796b1f09e7b470fb251ac2c88abd264/users/58eaf17b46004cdfbf66465f1775c605/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:28.154247 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44da61bd-c39c-487d-9ca9-d1202a264135 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e796b1f09e7b470fb251ac2c88abd264, user_id=58eaf17b46004cdfbf66465f1775c605, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:28.157520 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44da61bd-c39c-487d-9ca9-d1202a264135 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:28.188256 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 751/1521] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:28 2022] PUT /identity/v3/projects/e796b1f09e7b470fb251ac2c88abd264/users/58eaf17b46004cdfbf66465f1775c605/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:28.193527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b655e0df-7674-404d-a706-c91698022099 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:28.211873 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b655e0df-7674-404d-a706-c91698022099 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:28.331439 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e9603c-07b3-492c-93e1-dfa9e7b65ee7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:28.332092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e9603c-07b3-492c-93e1-dfa9e7b65ee7 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:28.332568 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e9603c-07b3-492c-93e1-dfa9e7b65ee7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:28.332997 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e9603c-07b3-492c-93e1-dfa9e7b65ee7 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:28.334234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e9603c-07b3-492c-93e1-dfa9e7b65ee7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:28.336272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e9603c-07b3-492c-93e1-dfa9e7b65ee7 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:28.350563 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 771/1522] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:28 2022] GET /identity/v3/roles => generated 1420 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:28.355283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44da61bd-c39c-487d-9ca9-d1202a264135 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:28.366729 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44da61bd-c39c-487d-9ca9-d1202a264135 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:28.375919 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dca5a40-c5e3-497c-88ee-f459a9c687d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:28.376597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dca5a40-c5e3-497c-88ee-f459a9c687d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:28.376876 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dca5a40-c5e3-497c-88ee-f459a9c687d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:28.377186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dca5a40-c5e3-497c-88ee-f459a9c687d1 admin admin] PATH_INFO: `/v3/projects/e796b1f09e7b470fb251ac2c88abd264/users/58eaf17b46004cdfbf66465f1775c605/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:28.379309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dca5a40-c5e3-497c-88ee-f459a9c687d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e796b1f09e7b470fb251ac2c88abd264, user_id=58eaf17b46004cdfbf66465f1775c605, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:28.384414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dca5a40-c5e3-497c-88ee-f459a9c687d1 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:28.395529 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 752/1523] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:28 2022] PUT /identity/v3/projects/e796b1f09e7b470fb251ac2c88abd264/users/58eaf17b46004cdfbf66465f1775c605/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:28.406642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8a4d7e-e884-41c3-b968-54266d05198a None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:28.407185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8a4d7e-e884-41c3-b968-54266d05198a None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:28.407281 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8a4d7e-e884-41c3-b968-54266d05198a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:28.438428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-5f8a4d7e-e884-41c3-b968-54266d05198a None None] MFA Rules not processed for user `58eaf17b46004cdfbf66465f1775c605`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:28.439672 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5f8a4d7e-e884-41c3-b968-54266d05198a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:28.570266 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 772/1524] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:28 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:28.596799 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dca5a40-c5e3-497c-88ee-f459a9c687d1 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:28.598654 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-7dca5a40-c5e3-497c-88ee-f459a9c687d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:28.685606 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dca5a40-c5e3-497c-88ee-f459a9c687d1 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:28.692148 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c710c21-cade-4be1-9394-55d52597f623 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:28.692865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c710c21-cade-4be1-9394-55d52597f623 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:28.693180 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c710c21-cade-4be1-9394-55d52597f623 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:28.693557 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c710c21-cade-4be1-9394-55d52597f623 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:28.704996 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c710c21-cade-4be1-9394-55d52597f623 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:28.707127 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c710c21-cade-4be1-9394-55d52597f623 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:28.720656 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 753/1525] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:28 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:28.910434 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8a4d7e-e884-41c3-b968-54266d05198a None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:28.922896 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8a4d7e-e884-41c3-b968-54266d05198a None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:28.927591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11b9c38-8bea-421c-b775-65b796686b85 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:28.928322 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11b9c38-8bea-421c-b775-65b796686b85 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:28.928750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11b9c38-8bea-421c-b775-65b796686b85 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:28.929102 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11b9c38-8bea-421c-b775-65b796686b85 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:28.993812 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11b9c38-8bea-421c-b775-65b796686b85 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:28.996308 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11b9c38-8bea-421c-b775-65b796686b85 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:29.000203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:29.008689 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 773/1526] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:28 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:33.515660 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c710c21-cade-4be1-9394-55d52597f623 service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:33.518083 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-9c710c21-cade-4be1-9394-55d52597f623 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:33.590685 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c710c21-cade-4be1-9394-55d52597f623 service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:33.595285 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02af7f36-d984-4eaf-8d06-dee31ef25753 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:33.595930 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02af7f36-d984-4eaf-8d06-dee31ef25753 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:33.596216 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02af7f36-d984-4eaf-8d06-dee31ef25753 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:33.596493 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02af7f36-d984-4eaf-8d06-dee31ef25753 admin admin] PATH_INFO: `/v3/users/e01d58067e2546b9bf8502e7fa729154` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:33.597627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02af7f36-d984-4eaf-8d06-dee31ef25753 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e01d58067e2546b9bf8502e7fa729154)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:33.599576 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02af7f36-d984-4eaf-8d06-dee31ef25753 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:33.615234 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:33.652351 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-02af7f36-d984-4eaf-8d06-dee31ef25753 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e01d58067e2546b9bf8502e7fa729154'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:33.659621 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-02af7f36-d984-4eaf-8d06-dee31ef25753 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e01d58067e2546b9bf8502e7fa729154'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:33.660295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-02af7f36-d984-4eaf-8d06-dee31ef25753 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e01d58067e2546b9bf8502e7fa729154'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:33.660906 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-02af7f36-d984-4eaf-8d06-dee31ef25753 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e01d58067e2546b9bf8502e7fa729154'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:33.674232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-02af7f36-d984-4eaf-8d06-dee31ef25753 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e01d58067e2546b9bf8502e7fa729154'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:33.687515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 754/1527] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:33 2022] DELETE /identity/v3/users/e01d58067e2546b9bf8502e7fa729154 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:33.691940 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b6b962-4333-412f-9f15-47d10384a0e2 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:33.692395 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b6b962-4333-412f-9f15-47d10384a0e2 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:33.692821 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b6b962-4333-412f-9f15-47d10384a0e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:33.712418 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-b7b6b962-4333-412f-9f15-47d10384a0e2 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:33.714353 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b7b6b962-4333-412f-9f15-47d10384a0e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:33.776448 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 774/1528] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:33.786173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02af7f36-d984-4eaf-8d06-dee31ef25753 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:33.787626 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-02af7f36-d984-4eaf-8d06-dee31ef25753 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:33.847027 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02af7f36-d984-4eaf-8d06-dee31ef25753 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:33.850556 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f29b408-204b-4d89-8379-7baf07963a4e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:33.851157 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29b408-204b-4d89-8379-7baf07963a4e service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:33.851436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29b408-204b-4d89-8379-7baf07963a4e service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:33.851743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29b408-204b-4d89-8379-7baf07963a4e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:33.861615 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f29b408-204b-4d89-8379-7baf07963a4e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:33.863058 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f29b408-204b-4d89-8379-7baf07963a4e service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:33.872912 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 755/1529] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:34.178269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b6b962-4333-412f-9f15-47d10384a0e2 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:34.179712 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b7b6b962-4333-412f-9f15-47d10384a0e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:34.192614 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b6b962-4333-412f-9f15-47d10384a0e2 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:34.198858 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba049e5f-8886-4a23-9a9a-d922c81aea40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:34.199748 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba049e5f-8886-4a23-9a9a-d922c81aea40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:34.200125 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba049e5f-8886-4a23-9a9a-d922c81aea40 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:34.200499 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba049e5f-8886-4a23-9a9a-d922c81aea40 admin admin] PATH_INFO: `/v3/projects/db9844d550444cf68b91c651487897fe` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:34.201743 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba049e5f-8886-4a23-9a9a-d922c81aea40 admin admin] RBAC: Authorizing `identity:delete_project(project_id=db9844d550444cf68b91c651487897fe)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:34.203865 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba049e5f-8886-4a23-9a9a-d922c81aea40 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:34.257960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ba049e5f-8886-4a23-9a9a-d922c81aea40 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db9844d550444cf68b91c651487897fe', 'request_id': 'req-ba049e5f-8886-4a23-9a9a-d922c81aea40'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:34.264062 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ba049e5f-8886-4a23-9a9a-d922c81aea40 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db9844d550444cf68b91c651487897fe', 'request_id': 'req-ba049e5f-8886-4a23-9a9a-d922c81aea40'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:34.270324 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ba049e5f-8886-4a23-9a9a-d922c81aea40 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db9844d550444cf68b91c651487897fe', 'request_id': 'req-ba049e5f-8886-4a23-9a9a-d922c81aea40'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:34.285274 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ba049e5f-8886-4a23-9a9a-d922c81aea40 admin admin] The token cache is being invalidate because project db9844d550444cf68b91c651487897fe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:34.285567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ba049e5f-8886-4a23-9a9a-d922c81aea40 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:34.286202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ba049e5f-8886-4a23-9a9a-d922c81aea40 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:34.287809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 775/1530] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:34 2022] DELETE /identity/v3/projects/db9844d550444cf68b91c651487897fe => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:34.293282 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f29b408-204b-4d89-8379-7baf07963a4e service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:34.303066 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f29b408-204b-4d89-8379-7baf07963a4e service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:34.369264 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f24ede8-d532-4bc6-933f-ea28e495336f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:34.370450 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f24ede8-d532-4bc6-933f-ea28e495336f admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:34.370804 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f24ede8-d532-4bc6-933f-ea28e495336f admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:34.371152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f24ede8-d532-4bc6-933f-ea28e495336f admin admin] PATH_INFO: `/v3/users/c779838194b9469396a6bf0b3a502dd0` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:34.372422 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f24ede8-d532-4bc6-933f-ea28e495336f admin admin] RBAC: Authorizing `identity:delete_user(user_id=c779838194b9469396a6bf0b3a502dd0)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:34.374420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f24ede8-d532-4bc6-933f-ea28e495336f admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:34.414244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:34.425052 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-4f24ede8-d532-4bc6-933f-ea28e495336f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c779838194b9469396a6bf0b3a502dd0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:34.433315 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-4f24ede8-d532-4bc6-933f-ea28e495336f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c779838194b9469396a6bf0b3a502dd0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:34.434228 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-4f24ede8-d532-4bc6-933f-ea28e495336f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c779838194b9469396a6bf0b3a502dd0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:34.435289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-4f24ede8-d532-4bc6-933f-ea28e495336f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c779838194b9469396a6bf0b3a502dd0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:34.448089 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-4f24ede8-d532-4bc6-933f-ea28e495336f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c779838194b9469396a6bf0b3a502dd0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:34.461228 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 756/1531] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:34 2022] DELETE /identity/v3/users/c779838194b9469396a6bf0b3a502dd0 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:34.737420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba049e5f-8886-4a23-9a9a-d922c81aea40 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:34.739003 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ba049e5f-8886-4a23-9a9a-d922c81aea40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:34.754742 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba049e5f-8886-4a23-9a9a-d922c81aea40 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:34.817029 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d17814-0c50-4f7e-958d-c6421e8b4102 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:34.817733 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d17814-0c50-4f7e-958d-c6421e8b4102 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:34.818016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d17814-0c50-4f7e-958d-c6421e8b4102 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:34.818286 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d17814-0c50-4f7e-958d-c6421e8b4102 admin admin] PATH_INFO: `/v3/projects/bb6becc00b6f414e92d6415a4df501e7` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:34.819411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d17814-0c50-4f7e-958d-c6421e8b4102 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb6becc00b6f414e92d6415a4df501e7)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:34.821361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d17814-0c50-4f7e-958d-c6421e8b4102 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:34.847926 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:34.871906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-43d17814-0c50-4f7e-958d-c6421e8b4102 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb6becc00b6f414e92d6415a4df501e7', 'request_id': 'req-43d17814-0c50-4f7e-958d-c6421e8b4102'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:34.877634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-43d17814-0c50-4f7e-958d-c6421e8b4102 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb6becc00b6f414e92d6415a4df501e7', 'request_id': 'req-43d17814-0c50-4f7e-958d-c6421e8b4102'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:34.884222 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-43d17814-0c50-4f7e-958d-c6421e8b4102 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb6becc00b6f414e92d6415a4df501e7', 'request_id': 'req-43d17814-0c50-4f7e-958d-c6421e8b4102'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:34.897162 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-43d17814-0c50-4f7e-958d-c6421e8b4102 admin admin] The token cache is being invalidate because project bb6becc00b6f414e92d6415a4df501e7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:34.897452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-43d17814-0c50-4f7e-958d-c6421e8b4102 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:34.898077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-43d17814-0c50-4f7e-958d-c6421e8b4102 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:34.899601 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 776/1532] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:34 2022] DELETE /identity/v3/projects/bb6becc00b6f414e92d6415a4df501e7 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:34.904988 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a337275a-b8c6-4329-b69e-31cef5787035 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:34.905460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a337275a-b8c6-4329-b69e-31cef5787035 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:34.905570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a337275a-b8c6-4329-b69e-31cef5787035 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:34.929692 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-a337275a-b8c6-4329-b69e-31cef5787035 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:34.930973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a337275a-b8c6-4329-b69e-31cef5787035 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:34.989918 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 757/1533] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:57:34.998338 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbdfc69-5294-4159-a609-4f4192f1250d None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:34.998617 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbdfc69-5294-4159-a609-4f4192f1250d None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:34.998887 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbdfc69-5294-4159-a609-4f4192f1250d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:35.018826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-5cbdfc69-5294-4159-a609-4f4192f1250d None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:35.020044 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5cbdfc69-5294-4159-a609-4f4192f1250d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:35.027751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 777/1534] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:35.031968 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a337275a-b8c6-4329-b69e-31cef5787035 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:35.041728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a337275a-b8c6-4329-b69e-31cef5787035 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:35.046692 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574f66c3-515e-4d1a-a987-0b0c937008df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:35.047314 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574f66c3-515e-4d1a-a987-0b0c937008df admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:35.047600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574f66c3-515e-4d1a-a987-0b0c937008df admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:35.047877 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574f66c3-515e-4d1a-a987-0b0c937008df admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:35.048522 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574f66c3-515e-4d1a-a987-0b0c937008df admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:57:35.048826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574f66c3-515e-4d1a-a987-0b0c937008df admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:35.050509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574f66c3-515e-4d1a-a987-0b0c937008df admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:35.062659 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 758/1535] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:57:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:57:35.066862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbdfc69-5294-4159-a609-4f4192f1250d None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:35.076593 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbdfc69-5294-4159-a609-4f4192f1250d None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:35.081197 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0f1b9b-5794-4a34-9e00-a033b52c3abd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:35.081785 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f1b9b-5794-4a34-9e00-a033b52c3abd admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:35.082065 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f1b9b-5794-4a34-9e00-a033b52c3abd admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:35.082341 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f1b9b-5794-4a34-9e00-a033b52c3abd admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:35.083220 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0f1b9b-5794-4a34-9e00-a033b52c3abd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:35.085175 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0f1b9b-5794-4a34-9e00-a033b52c3abd admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:35.111324 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 778/1536] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:35 2022] POST /identity/v3/projects => generated 363 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:35.115418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574f66c3-515e-4d1a-a987-0b0c937008df admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:35.125879 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574f66c3-515e-4d1a-a987-0b0c937008df admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:35.180702 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad5653d-6dd9-41f1-a778-9ad145cd8532 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:35.181298 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad5653d-6dd9-41f1-a778-9ad145cd8532 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:35.181576 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad5653d-6dd9-41f1-a778-9ad145cd8532 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:35.182363 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad5653d-6dd9-41f1-a778-9ad145cd8532 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:35.183537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad5653d-6dd9-41f1-a778-9ad145cd8532 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:35.184971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad5653d-6dd9-41f1-a778-9ad145cd8532 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:35.207768 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 759/1537] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:35 2022] POST /identity/v3/users => generated 282 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:35.212065 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0f1b9b-5794-4a34-9e00-a033b52c3abd admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:35.223285 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0f1b9b-5794-4a34-9e00-a033b52c3abd admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:35.228127 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81caf9e-d7df-4b80-8ae7-018e23e22cfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:35.228834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81caf9e-d7df-4b80-8ae7-018e23e22cfb admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:35.229216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81caf9e-d7df-4b80-8ae7-018e23e22cfb admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:35.229486 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81caf9e-d7df-4b80-8ae7-018e23e22cfb admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:35.230123 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81caf9e-d7df-4b80-8ae7-018e23e22cfb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:35.231772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81caf9e-d7df-4b80-8ae7-018e23e22cfb admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:35.242033 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 779/1538] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:35.246669 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad5653d-6dd9-41f1-a778-9ad145cd8532 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:35.257088 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad5653d-6dd9-41f1-a778-9ad145cd8532 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:35.263312 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876204f0-10f6-4037-b860-f81f2c52016e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:35.263489 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876204f0-10f6-4037-b860-f81f2c52016e admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:35.263771 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876204f0-10f6-4037-b860-f81f2c52016e admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:35.264076 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876204f0-10f6-4037-b860-f81f2c52016e admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:35.264770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876204f0-10f6-4037-b860-f81f2c52016e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:35.266329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876204f0-10f6-4037-b860-f81f2c52016e admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:35.284273 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 760/1539] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:35 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:35.285421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81caf9e-d7df-4b80-8ae7-018e23e22cfb admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:35.295449 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81caf9e-d7df-4b80-8ae7-018e23e22cfb admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:35.300059 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256738bf-f9aa-4b66-991b-36470c7e0d9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:35.300705 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256738bf-f9aa-4b66-991b-36470c7e0d9f admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:35.300981 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256738bf-f9aa-4b66-991b-36470c7e0d9f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:35.301251 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256738bf-f9aa-4b66-991b-36470c7e0d9f admin admin] PATH_INFO: `/v3/projects/6192625f547e4870bca3dc6554aa41da/users/89780bb57cf84179a06e20983027b86a/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:35.317156 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256738bf-f9aa-4b66-991b-36470c7e0d9f admin admin] RBAC: Authorizing `identity:create_grant(project_id=6192625f547e4870bca3dc6554aa41da, user_id=89780bb57cf84179a06e20983027b86a, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:35.319927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256738bf-f9aa-4b66-991b-36470c7e0d9f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:35.340990 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 780/1540] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:35 2022] PUT /identity/v3/projects/6192625f547e4870bca3dc6554aa41da/users/89780bb57cf84179a06e20983027b86a/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:35.345539 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d997e1-0359-4628-99a6-1d155759383e None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:35.345897 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d997e1-0359-4628-99a6-1d155759383e None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:35.346323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d997e1-0359-4628-99a6-1d155759383e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:35.375808 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-93d997e1-0359-4628-99a6-1d155759383e None None] MFA Rules not processed for user `89780bb57cf84179a06e20983027b86a`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:35.377071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-93d997e1-0359-4628-99a6-1d155759383e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:35.385010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:35.446428 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 761/1541] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:35 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:57:35.450923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256738bf-f9aa-4b66-991b-36470c7e0d9f admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:35.537333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256738bf-f9aa-4b66-991b-36470c7e0d9f admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:35.542792 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c6e520-5432-4ee5-bbc9-ff2f7e0b6ef8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:35.543415 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c6e520-5432-4ee5-bbc9-ff2f7e0b6ef8 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:35.543716 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c6e520-5432-4ee5-bbc9-ff2f7e0b6ef8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:35.543989 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c6e520-5432-4ee5-bbc9-ff2f7e0b6ef8 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:35.544638 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c6e520-5432-4ee5-bbc9-ff2f7e0b6ef8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:35.546620 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c6e520-5432-4ee5-bbc9-ff2f7e0b6ef8 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:35.576971 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 781/1542] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:35 2022] POST /identity/v3/projects => generated 365 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:57:35.581142 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d997e1-0359-4628-99a6-1d155759383e None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:35.593575 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d997e1-0359-4628-99a6-1d155759383e None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:35.670310 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b514b98-1475-4ae1-841d-74f7bbe7d21a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:35.670940 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b514b98-1475-4ae1-841d-74f7bbe7d21a admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:35.671312 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b514b98-1475-4ae1-841d-74f7bbe7d21a admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:35.671705 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b514b98-1475-4ae1-841d-74f7bbe7d21a admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:35.672613 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b514b98-1475-4ae1-841d-74f7bbe7d21a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:35.674613 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b514b98-1475-4ae1-841d-74f7bbe7d21a admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:35.716649 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c6e520-5432-4ee5-bbc9-ff2f7e0b6ef8 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:35.720338 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 762/1543] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:35 2022] POST /identity/v3/users => generated 289 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:35.761704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c6e520-5432-4ee5-bbc9-ff2f7e0b6ef8 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:35.773822 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b2cdc7-0a66-4a13-92fa-6580111c6153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:35.774466 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b2cdc7-0a66-4a13-92fa-6580111c6153 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:35.774969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b2cdc7-0a66-4a13-92fa-6580111c6153 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:35.775881 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b2cdc7-0a66-4a13-92fa-6580111c6153 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:35.777739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b2cdc7-0a66-4a13-92fa-6580111c6153 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:35.779772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b2cdc7-0a66-4a13-92fa-6580111c6153 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:35.795969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 782/1544] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:35 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:35.798173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b514b98-1475-4ae1-841d-74f7bbe7d21a admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:35.809752 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b514b98-1475-4ae1-841d-74f7bbe7d21a admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:35.816884 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daaad50f-e142-4dd0-9483-9e666201fc50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:35.816884 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daaad50f-e142-4dd0-9483-9e666201fc50 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:35.817162 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daaad50f-e142-4dd0-9483-9e666201fc50 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:35.817348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daaad50f-e142-4dd0-9483-9e666201fc50 admin admin] PATH_INFO: `/v3/projects/577d026033e64a8da7b66d1d1061fd27/users/cd67b698f76048c79c2a240a60b950fb/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:35.837514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daaad50f-e142-4dd0-9483-9e666201fc50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=577d026033e64a8da7b66d1d1061fd27, user_id=cd67b698f76048c79c2a240a60b950fb, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:35.843073 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daaad50f-e142-4dd0-9483-9e666201fc50 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:35.867579 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 763/1545] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:35 2022] PUT /identity/v3/projects/577d026033e64a8da7b66d1d1061fd27/users/cd67b698f76048c79c2a240a60b950fb/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:35.872593 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b2cdc7-0a66-4a13-92fa-6580111c6153 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:35.886833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b2cdc7-0a66-4a13-92fa-6580111c6153 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:35.935771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=90,keystone:DELETE=6,keystone:INSERT=4 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:36.027599 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b5110d-1b59-4443-b665-a47d40d45136 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:36.028234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b5110d-1b59-4443-b665-a47d40d45136 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:36.028687 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b5110d-1b59-4443-b665-a47d40d45136 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:36.029023 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b5110d-1b59-4443-b665-a47d40d45136 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:36.029744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b5110d-1b59-4443-b665-a47d40d45136 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:36.031445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b5110d-1b59-4443-b665-a47d40d45136 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:36.045027 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 783/1546] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:35 2022] GET /identity/v3/roles => generated 1420 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:36.049436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daaad50f-e142-4dd0-9483-9e666201fc50 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:36.064194 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daaad50f-e142-4dd0-9483-9e666201fc50 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:36.071881 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9602263f-5ffa-4d1d-8ac2-05b417a8eedb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:36.079735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9602263f-5ffa-4d1d-8ac2-05b417a8eedb admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:36.079735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9602263f-5ffa-4d1d-8ac2-05b417a8eedb admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:36.079735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9602263f-5ffa-4d1d-8ac2-05b417a8eedb admin admin] PATH_INFO: `/v3/projects/577d026033e64a8da7b66d1d1061fd27/users/cd67b698f76048c79c2a240a60b950fb/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:36.079735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9602263f-5ffa-4d1d-8ac2-05b417a8eedb admin admin] RBAC: Authorizing `identity:create_grant(project_id=577d026033e64a8da7b66d1d1061fd27, user_id=cd67b698f76048c79c2a240a60b950fb, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:36.079735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9602263f-5ffa-4d1d-8ac2-05b417a8eedb admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:36.097698 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 764/1547] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:36 2022] PUT /identity/v3/projects/577d026033e64a8da7b66d1d1061fd27/users/cd67b698f76048c79c2a240a60b950fb/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:36.110620 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eec7d1b-c5e2-4fdc-9c94-7f592dcbb37f None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:36.111442 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eec7d1b-c5e2-4fdc-9c94-7f592dcbb37f None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:36.112131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eec7d1b-c5e2-4fdc-9c94-7f592dcbb37f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:36.155063 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-0eec7d1b-c5e2-4fdc-9c94-7f592dcbb37f None None] MFA Rules not processed for user `cd67b698f76048c79c2a240a60b950fb`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:36.156491 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0eec7d1b-c5e2-4fdc-9c94-7f592dcbb37f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:36.303395 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 784/1548] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:36 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:36.327736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9602263f-5ffa-4d1d-8ac2-05b417a8eedb admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:36.329840 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-9602263f-5ffa-4d1d-8ac2-05b417a8eedb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:36.397127 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9602263f-5ffa-4d1d-8ac2-05b417a8eedb admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:36.401954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d90284-7876-47f7-8f13-dd6f3479e768 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:36.402664 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d90284-7876-47f7-8f13-dd6f3479e768 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:36.403018 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d90284-7876-47f7-8f13-dd6f3479e768 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:36.403367 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d90284-7876-47f7-8f13-dd6f3479e768 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:36.419002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d90284-7876-47f7-8f13-dd6f3479e768 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:36.421860 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d90284-7876-47f7-8f13-dd6f3479e768 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:36.438790 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 765/1549] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:36 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:37.187096 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eec7d1b-c5e2-4fdc-9c94-7f592dcbb37f None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:37.258285 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eec7d1b-c5e2-4fdc-9c94-7f592dcbb37f None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:37.263240 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f764fe-1196-42fd-8077-ee9168378968 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:37.263908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f764fe-1196-42fd-8077-ee9168378968 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:37.264188 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f764fe-1196-42fd-8077-ee9168378968 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:37.264573 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f764fe-1196-42fd-8077-ee9168378968 admin admin] PATH_INFO: `/v3/users/89780bb57cf84179a06e20983027b86a` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:37.265925 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f764fe-1196-42fd-8077-ee9168378968 admin admin] RBAC: Authorizing `identity:delete_user(user_id=89780bb57cf84179a06e20983027b86a)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:37.267936 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f764fe-1196-42fd-8077-ee9168378968 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:37.330363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:37.335877 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-53f764fe-1196-42fd-8077-ee9168378968 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89780bb57cf84179a06e20983027b86a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:37.337019 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-53f764fe-1196-42fd-8077-ee9168378968 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89780bb57cf84179a06e20983027b86a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:37.364258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-53f764fe-1196-42fd-8077-ee9168378968 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89780bb57cf84179a06e20983027b86a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:37.378561 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-53f764fe-1196-42fd-8077-ee9168378968 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89780bb57cf84179a06e20983027b86a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:37.399930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-53f764fe-1196-42fd-8077-ee9168378968 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89780bb57cf84179a06e20983027b86a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:37.409122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 785/1550] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:37 2022] DELETE /identity/v3/users/89780bb57cf84179a06e20983027b86a => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:37.413337 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9bfed0-c56e-4555-a4f9-5964b1e0dc4c None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:37.413813 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9bfed0-c56e-4555-a4f9-5964b1e0dc4c None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:37.413946 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9bfed0-c56e-4555-a4f9-5964b1e0dc4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:37.425694 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:37.438003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-ae9bfed0-c56e-4555-a4f9-5964b1e0dc4c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:37.439225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ae9bfed0-c56e-4555-a4f9-5964b1e0dc4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:37.499807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 766/1551] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:37.509341 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f764fe-1196-42fd-8077-ee9168378968 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:37.510752 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-53f764fe-1196-42fd-8077-ee9168378968 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:37.568205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f764fe-1196-42fd-8077-ee9168378968 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:37.571726 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aef9bf8-0cae-47f4-a932-a93e9d884035 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:37.572375 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aef9bf8-0cae-47f4-a932-a93e9d884035 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:37.572785 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aef9bf8-0cae-47f4-a932-a93e9d884035 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:37.573054 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aef9bf8-0cae-47f4-a932-a93e9d884035 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:37.582162 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aef9bf8-0cae-47f4-a932-a93e9d884035 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:37.583866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aef9bf8-0cae-47f4-a932-a93e9d884035 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:37.594525 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 786/1552] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:37.984214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9bfed0-c56e-4555-a4f9-5964b1e0dc4c None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:37.986524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ae9bfed0-c56e-4555-a4f9-5964b1e0dc4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:38.001751 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9bfed0-c56e-4555-a4f9-5964b1e0dc4c None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:38.007034 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12a55d5-d20e-4ede-80bb-402c4b2842a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:38.007870 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12a55d5-d20e-4ede-80bb-402c4b2842a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:38.008229 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12a55d5-d20e-4ede-80bb-402c4b2842a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:38.008704 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12a55d5-d20e-4ede-80bb-402c4b2842a8 admin admin] PATH_INFO: `/v3/projects/6192625f547e4870bca3dc6554aa41da` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:38.011653 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12a55d5-d20e-4ede-80bb-402c4b2842a8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6192625f547e4870bca3dc6554aa41da)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:38.014155 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12a55d5-d20e-4ede-80bb-402c4b2842a8 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:38.079645 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f12a55d5-d20e-4ede-80bb-402c4b2842a8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6192625f547e4870bca3dc6554aa41da', 'request_id': 'req-f12a55d5-d20e-4ede-80bb-402c4b2842a8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:38.090862 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f12a55d5-d20e-4ede-80bb-402c4b2842a8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6192625f547e4870bca3dc6554aa41da', 'request_id': 'req-f12a55d5-d20e-4ede-80bb-402c4b2842a8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:38.099592 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f12a55d5-d20e-4ede-80bb-402c4b2842a8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6192625f547e4870bca3dc6554aa41da', 'request_id': 'req-f12a55d5-d20e-4ede-80bb-402c4b2842a8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:38.125204 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f12a55d5-d20e-4ede-80bb-402c4b2842a8 admin admin] The token cache is being invalidate because project 6192625f547e4870bca3dc6554aa41da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:38.126309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f12a55d5-d20e-4ede-80bb-402c4b2842a8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:38.128649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f12a55d5-d20e-4ede-80bb-402c4b2842a8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:38.131322 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 767/1553] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:37 2022] DELETE /identity/v3/projects/6192625f547e4870bca3dc6554aa41da => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:38.136518 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aef9bf8-0cae-47f4-a932-a93e9d884035 service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:38.148384 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aef9bf8-0cae-47f4-a932-a93e9d884035 service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:38.223899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb975d89-996e-43eb-954b-1e4136147bf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:38.224668 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb975d89-996e-43eb-954b-1e4136147bf9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:38.225027 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb975d89-996e-43eb-954b-1e4136147bf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:38.225368 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb975d89-996e-43eb-954b-1e4136147bf9 admin admin] PATH_INFO: `/v3/users/cd67b698f76048c79c2a240a60b950fb` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:38.226580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb975d89-996e-43eb-954b-1e4136147bf9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cd67b698f76048c79c2a240a60b950fb)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:38.229302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb975d89-996e-43eb-954b-1e4136147bf9 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:38.235766 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12a55d5-d20e-4ede-80bb-402c4b2842a8 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:38.237243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-f12a55d5-d20e-4ede-80bb-402c4b2842a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:38.250927 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12a55d5-d20e-4ede-80bb-402c4b2842a8 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:38.255981 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db939e1-6216-4782-a0cf-3041e97d1f47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:38.256639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db939e1-6216-4782-a0cf-3041e97d1f47 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:38.256901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db939e1-6216-4782-a0cf-3041e97d1f47 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:38.257178 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db939e1-6216-4782-a0cf-3041e97d1f47 admin admin] PATH_INFO: `/v3/users/b0a4f3bff51b4304bcfe4434db9c0f76` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:38.258367 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db939e1-6216-4782-a0cf-3041e97d1f47 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b0a4f3bff51b4304bcfe4434db9c0f76)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:38.260968 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db939e1-6216-4782-a0cf-3041e97d1f47 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:38.287193 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-bb975d89-996e-43eb-954b-1e4136147bf9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cd67b698f76048c79c2a240a60b950fb'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:38.287866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-bb975d89-996e-43eb-954b-1e4136147bf9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cd67b698f76048c79c2a240a60b950fb'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:38.300239 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-bb975d89-996e-43eb-954b-1e4136147bf9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cd67b698f76048c79c2a240a60b950fb'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:38.303143 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:38.310520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-bb975d89-996e-43eb-954b-1e4136147bf9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cd67b698f76048c79c2a240a60b950fb'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:38.319746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-bb975d89-996e-43eb-954b-1e4136147bf9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cd67b698f76048c79c2a240a60b950fb'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:38.327835 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-9db939e1-6216-4782-a0cf-3041e97d1f47 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b0a4f3bff51b4304bcfe4434db9c0f76'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:38.331442 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 787/1554] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:38 2022] DELETE /identity/v3/users/cd67b698f76048c79c2a240a60b950fb => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:38.335416 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-9db939e1-6216-4782-a0cf-3041e97d1f47 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b0a4f3bff51b4304bcfe4434db9c0f76'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:38.336180 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-9db939e1-6216-4782-a0cf-3041e97d1f47 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b0a4f3bff51b4304bcfe4434db9c0f76'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:38.336821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-9db939e1-6216-4782-a0cf-3041e97d1f47 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b0a4f3bff51b4304bcfe4434db9c0f76'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:38.349462 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-9db939e1-6216-4782-a0cf-3041e97d1f47 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b0a4f3bff51b4304bcfe4434db9c0f76'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:38.360310 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:38.368298 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 768/1555] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:38 2022] DELETE /identity/v3/users/b0a4f3bff51b4304bcfe4434db9c0f76 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:38.371104 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1977ff31-86f1-41fa-b4a1-352a6de1b242 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:38.371476 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1977ff31-86f1-41fa-b4a1-352a6de1b242 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:38.371842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1977ff31-86f1-41fa-b4a1-352a6de1b242 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:38.392118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-1977ff31-86f1-41fa-b4a1-352a6de1b242 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:38.393340 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1977ff31-86f1-41fa-b4a1-352a6de1b242 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:38.472604 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 788/1556] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:57:38.484982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db939e1-6216-4782-a0cf-3041e97d1f47 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:38.487336 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-9db939e1-6216-4782-a0cf-3041e97d1f47 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:38.563966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db939e1-6216-4782-a0cf-3041e97d1f47 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:38.567836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f37f9f-7eec-45c3-9b37-9c0163861424 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:38.568590 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f37f9f-7eec-45c3-9b37-9c0163861424 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:38.568876 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f37f9f-7eec-45c3-9b37-9c0163861424 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:38.569149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f37f9f-7eec-45c3-9b37-9c0163861424 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:38.579525 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f37f9f-7eec-45c3-9b37-9c0163861424 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:38.581042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f37f9f-7eec-45c3-9b37-9c0163861424 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:38.592136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 769/1557] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:39.001273 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1977ff31-86f1-41fa-b4a1-352a6de1b242 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:39.003050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1977ff31-86f1-41fa-b4a1-352a6de1b242 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:39.018846 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1977ff31-86f1-41fa-b4a1-352a6de1b242 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:39.027047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a704685f-671a-47ef-baf2-12956800d28b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:39.028074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a704685f-671a-47ef-baf2-12956800d28b admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:39.028407 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a704685f-671a-47ef-baf2-12956800d28b admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:39.028673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a704685f-671a-47ef-baf2-12956800d28b admin admin] PATH_INFO: `/v3/projects/7308e9a0e8b24dd68e7031770f9bd368` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:39.030446 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a704685f-671a-47ef-baf2-12956800d28b admin admin] RBAC: Authorizing `identity:delete_project(project_id=7308e9a0e8b24dd68e7031770f9bd368)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:39.034177 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a704685f-671a-47ef-baf2-12956800d28b admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:39.085462 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f37f9f-7eec-45c3-9b37-9c0163861424 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:39.089206 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-25f37f9f-7eec-45c3-9b37-9c0163861424 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:39.124350 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f37f9f-7eec-45c3-9b37-9c0163861424 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:39.131649 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a704685f-671a-47ef-baf2-12956800d28b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7308e9a0e8b24dd68e7031770f9bd368', 'request_id': 'req-a704685f-671a-47ef-baf2-12956800d28b'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:39.140393 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a704685f-671a-47ef-baf2-12956800d28b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7308e9a0e8b24dd68e7031770f9bd368', 'request_id': 'req-a704685f-671a-47ef-baf2-12956800d28b'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:39.149648 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a704685f-671a-47ef-baf2-12956800d28b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7308e9a0e8b24dd68e7031770f9bd368', 'request_id': 'req-a704685f-671a-47ef-baf2-12956800d28b'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:39.175339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a704685f-671a-47ef-baf2-12956800d28b admin admin] The token cache is being invalidate because project 7308e9a0e8b24dd68e7031770f9bd368 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:39.176041 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a704685f-671a-47ef-baf2-12956800d28b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:39.176925 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a704685f-671a-47ef-baf2-12956800d28b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:39.182847 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 789/1558] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:38 2022] DELETE /identity/v3/projects/7308e9a0e8b24dd68e7031770f9bd368 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:39.183909 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a704685f-671a-47ef-baf2-12956800d28b admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:39.186645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a704685f-671a-47ef-baf2-12956800d28b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:39.216570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db70f06-ecf4-4240-904a-b2474a40dd18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:39.217545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db70f06-ecf4-4240-904a-b2474a40dd18 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:39.217923 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db70f06-ecf4-4240-904a-b2474a40dd18 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:39.219419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db70f06-ecf4-4240-904a-b2474a40dd18 admin admin] PATH_INFO: `/v3/projects/577d026033e64a8da7b66d1d1061fd27` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:39.221145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db70f06-ecf4-4240-904a-b2474a40dd18 admin admin] RBAC: Authorizing `identity:delete_project(project_id=577d026033e64a8da7b66d1d1061fd27)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:39.224647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db70f06-ecf4-4240-904a-b2474a40dd18 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:39.304381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-4db70f06-ecf4-4240-904a-b2474a40dd18 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '577d026033e64a8da7b66d1d1061fd27', 'request_id': 'req-4db70f06-ecf4-4240-904a-b2474a40dd18'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:39.330081 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-4db70f06-ecf4-4240-904a-b2474a40dd18 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '577d026033e64a8da7b66d1d1061fd27', 'request_id': 'req-4db70f06-ecf4-4240-904a-b2474a40dd18'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:39.335709 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a704685f-671a-47ef-baf2-12956800d28b admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:39.344718 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-4db70f06-ecf4-4240-904a-b2474a40dd18 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '577d026033e64a8da7b66d1d1061fd27', 'request_id': 'req-4db70f06-ecf4-4240-904a-b2474a40dd18'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:39.356500 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:39.362487 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-4db70f06-ecf4-4240-904a-b2474a40dd18 admin admin] The token cache is being invalidate because project 577d026033e64a8da7b66d1d1061fd27 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:39.362930 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-4db70f06-ecf4-4240-904a-b2474a40dd18 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:39.363639 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-4db70f06-ecf4-4240-904a-b2474a40dd18 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:39.365933 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 770/1559] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:39 2022] DELETE /identity/v3/projects/577d026033e64a8da7b66d1d1061fd27 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:39.367431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d90684-192b-43f7-a9d4-c1175fd6f4c1 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:39.367923 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d90684-192b-43f7-a9d4-c1175fd6f4c1 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:39.368204 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d90684-192b-43f7-a9d4-c1175fd6f4c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:39.397660 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-b7d90684-192b-43f7-a9d4-c1175fd6f4c1 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:39.398920 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-b7d90684-192b-43f7-a9d4-c1175fd6f4c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:39.424719 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffe5cd4-3b30-41b9-8656-a179dc8a2314 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:39.425461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffe5cd4-3b30-41b9-8656-a179dc8a2314 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:39.425864 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffe5cd4-3b30-41b9-8656-a179dc8a2314 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:39.426202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffe5cd4-3b30-41b9-8656-a179dc8a2314 admin admin] PATH_INFO: `/v3/users/0efa2d3ea582464398471fa1780c4370` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:39.427956 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffe5cd4-3b30-41b9-8656-a179dc8a2314 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0efa2d3ea582464398471fa1780c4370)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:39.431032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffe5cd4-3b30-41b9-8656-a179dc8a2314 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:39.440243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:39.491164 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 771/1560] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:39.501291 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a39155-38c0-42b9-b8d1-6e594202c2db None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:39.501736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a39155-38c0-42b9-b8d1-6e594202c2db None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:39.502161 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a39155-38c0-42b9-b8d1-6e594202c2db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:39.502953 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-1ffe5cd4-3b30-41b9-8656-a179dc8a2314 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0efa2d3ea582464398471fa1780c4370'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:39.508100 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-1ffe5cd4-3b30-41b9-8656-a179dc8a2314 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0efa2d3ea582464398471fa1780c4370'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:39.525798 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-1ffe5cd4-3b30-41b9-8656-a179dc8a2314 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0efa2d3ea582464398471fa1780c4370'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:39.533272 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-b8a39155-38c0-42b9-b8d1-6e594202c2db None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:39.534948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-b8a39155-38c0-42b9-b8d1-6e594202c2db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:39.535647 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-1ffe5cd4-3b30-41b9-8656-a179dc8a2314 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0efa2d3ea582464398471fa1780c4370'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:39.546545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 772/1561] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:39.548263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d643b843-bd72-4db1-80d1-37ab2240f7a6 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:39.548620 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d643b843-bd72-4db1-80d1-37ab2240f7a6 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:39.549021 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d643b843-bd72-4db1-80d1-37ab2240f7a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:39.552230 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-1ffe5cd4-3b30-41b9-8656-a179dc8a2314 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0efa2d3ea582464398471fa1780c4370'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:39.560600 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 790/1562] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:39 2022] DELETE /identity/v3/users/0efa2d3ea582464398471fa1780c4370 => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:39.562173 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c225c528-ca26-4da3-8c2b-cf9ee616c30e None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:39.562525 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c225c528-ca26-4da3-8c2b-cf9ee616c30e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:39.562868 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c225c528-ca26-4da3-8c2b-cf9ee616c30e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:39.572493 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-d643b843-bd72-4db1-80d1-37ab2240f7a6 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:39.573706 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d643b843-bd72-4db1-80d1-37ab2240f7a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:39.639349 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 773/1563] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:39.640983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3320d6-b31d-4388-999e-0e705cc53f3d None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:39.641428 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3320d6-b31d-4388-999e-0e705cc53f3d None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:39.641852 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3320d6-b31d-4388-999e-0e705cc53f3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:39.663256 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-4a3320d6-b31d-4388-999e-0e705cc53f3d None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:39.664538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-4a3320d6-b31d-4388-999e-0e705cc53f3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:39.673557 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 774/1564] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:39.674433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3320d6-b31d-4388-999e-0e705cc53f3d None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:39.686629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3320d6-b31d-4388-999e-0e705cc53f3d None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:39.691289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f755f5-1f2e-4bd1-bee3-17c71fd3a0a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:39.691938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f755f5-1f2e-4bd1-bee3-17c71fd3a0a7 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:39.692221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f755f5-1f2e-4bd1-bee3-17c71fd3a0a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:39.692497 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f755f5-1f2e-4bd1-bee3-17c71fd3a0a7 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:39.693152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f755f5-1f2e-4bd1-bee3-17c71fd3a0a7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:57:39.693472 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f755f5-1f2e-4bd1-bee3-17c71fd3a0a7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:39.695061 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f755f5-1f2e-4bd1-bee3-17c71fd3a0a7 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:39.705919 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-c225c528-ca26-4da3-8c2b-cf9ee616c30e None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:39.707147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-c225c528-ca26-4da3-8c2b-cf9ee616c30e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:39.711223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 775/1565] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:57:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:57:39.711223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f755f5-1f2e-4bd1-bee3-17c71fd3a0a7 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:39.711223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-f4f755f5-1f2e-4bd1-bee3-17c71fd3a0a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:39.718513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 791/1566] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:39.719295 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c225c528-ca26-4da3-8c2b-cf9ee616c30e None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:39.729453 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c225c528-ca26-4da3-8c2b-cf9ee616c30e None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:39.734966 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3a21c9-e427-46b6-aadd-5569422edf9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:39.735589 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3a21c9-e427-46b6-aadd-5569422edf9b admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:39.735863 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3a21c9-e427-46b6-aadd-5569422edf9b admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:39.736132 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3a21c9-e427-46b6-aadd-5569422edf9b admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:39.736776 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3a21c9-e427-46b6-aadd-5569422edf9b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:39.739085 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3a21c9-e427-46b6-aadd-5569422edf9b admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:39.767807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 792/1567] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:39 2022] POST /identity/v3/projects => generated 383 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:39.768648 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3a21c9-e427-46b6-aadd-5569422edf9b admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:39.770503 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f755f5-1f2e-4bd1-bee3-17c71fd3a0a7 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:39.820744 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438da060-3d96-4aef-a36d-a705532d00d3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:39.821365 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438da060-3d96-4aef-a36d-a705532d00d3 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:39.821646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438da060-3d96-4aef-a36d-a705532d00d3 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:39.821920 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438da060-3d96-4aef-a36d-a705532d00d3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:39.830479 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3a21c9-e427-46b6-aadd-5569422edf9b admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:39.831767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438da060-3d96-4aef-a36d-a705532d00d3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:39.833477 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438da060-3d96-4aef-a36d-a705532d00d3 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:39.835299 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a146508f-4b4a-4757-9f2a-9dc8abcc03c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:39.835934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a146508f-4b4a-4757-9f2a-9dc8abcc03c5 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:39.836218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a146508f-4b4a-4757-9f2a-9dc8abcc03c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:39.836527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a146508f-4b4a-4757-9f2a-9dc8abcc03c5 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:39.837131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a146508f-4b4a-4757-9f2a-9dc8abcc03c5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:57:39.837466 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a146508f-4b4a-4757-9f2a-9dc8abcc03c5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:39.839712 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a146508f-4b4a-4757-9f2a-9dc8abcc03c5 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:39.844923 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 776/1568] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:39.845725 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438da060-3d96-4aef-a36d-a705532d00d3 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:39.851476 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 793/1569] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:57:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:57:39.852319 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:39.856725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a146508f-4b4a-4757-9f2a-9dc8abcc03c5 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:39.858010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438da060-3d96-4aef-a36d-a705532d00d3 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:39.862724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1176b0e1-4965-46fb-8e44-3e6bd68df0e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:39.863648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1176b0e1-4965-46fb-8e44-3e6bd68df0e2 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:39.864323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1176b0e1-4965-46fb-8e44-3e6bd68df0e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:39.864323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1176b0e1-4965-46fb-8e44-3e6bd68df0e2 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:39.865005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1176b0e1-4965-46fb-8e44-3e6bd68df0e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:39.867247 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1176b0e1-4965-46fb-8e44-3e6bd68df0e2 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:39.869378 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a146508f-4b4a-4757-9f2a-9dc8abcc03c5 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:39.875150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150c3a03-e9d6-445d-8e8a-790be9e803ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:39.875890 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150c3a03-e9d6-445d-8e8a-790be9e803ac admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:39.876239 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150c3a03-e9d6-445d-8e8a-790be9e803ac admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:39.876558 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150c3a03-e9d6-445d-8e8a-790be9e803ac admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:39.877269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150c3a03-e9d6-445d-8e8a-790be9e803ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:39.879021 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:39.879149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150c3a03-e9d6-445d-8e8a-790be9e803ac admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:39.898991 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 777/1570] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:39 2022] POST /identity/v3/users => generated 292 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:39.901149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1176b0e1-4965-46fb-8e44-3e6bd68df0e2 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:39.912054 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1176b0e1-4965-46fb-8e44-3e6bd68df0e2 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:39.918087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad89976-720c-4b5e-952a-363849a505a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:39.918870 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad89976-720c-4b5e-952a-363849a505a1 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:39.919148 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad89976-720c-4b5e-952a-363849a505a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:39.920006 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 794/1571] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:39 2022] POST /identity/v3/projects => generated 369 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:57:39.922516 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad89976-720c-4b5e-952a-363849a505a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:39.923226 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad89976-720c-4b5e-952a-363849a505a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:39.923466 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150c3a03-e9d6-445d-8e8a-790be9e803ac admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:39.924884 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad89976-720c-4b5e-952a-363849a505a1 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:39.937255 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 778/1572] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:39 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 14:57:39.940967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad89976-720c-4b5e-952a-363849a505a1 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:39.953855 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad89976-720c-4b5e-952a-363849a505a1 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:40.005060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150c3a03-e9d6-445d-8e8a-790be9e803ac admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:40.011374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05eff2cf-21d3-453b-9d9b-ceb8430158a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:40.012879 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eff2cf-21d3-453b-9d9b-ceb8430158a7 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:40.013301 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eff2cf-21d3-453b-9d9b-ceb8430158a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:40.013576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eff2cf-21d3-453b-9d9b-ceb8430158a7 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:40.014211 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05eff2cf-21d3-453b-9d9b-ceb8430158a7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:40.016250 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05eff2cf-21d3-453b-9d9b-ceb8430158a7 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:40.018983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b889323-ef8a-4fd8-b7ea-85263d13b0bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:40.019740 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b889323-ef8a-4fd8-b7ea-85263d13b0bc admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:40.023354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b889323-ef8a-4fd8-b7ea-85263d13b0bc admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:40.023354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b889323-ef8a-4fd8-b7ea-85263d13b0bc admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:40.023354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b889323-ef8a-4fd8-b7ea-85263d13b0bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:40.023354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b889323-ef8a-4fd8-b7ea-85263d13b0bc admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:40.041461 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 779/1573] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:39 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 14:57:40.043030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b889323-ef8a-4fd8-b7ea-85263d13b0bc admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:40.060569 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b889323-ef8a-4fd8-b7ea-85263d13b0bc admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:40.061080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 795/1574] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:39 2022] POST /identity/v3/users => generated 285 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:57:40.062509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05eff2cf-21d3-453b-9d9b-ceb8430158a7 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:40.065305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e0c230-3dd7-4d30-bf66-f5c069ad5cf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:40.066288 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e0c230-3dd7-4d30-bf66-f5c069ad5cf1 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:40.066406 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e0c230-3dd7-4d30-bf66-f5c069ad5cf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:40.066654 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e0c230-3dd7-4d30-bf66-f5c069ad5cf1 admin admin] PATH_INFO: `/v3/projects/5575957500cd4f84b4d65520cece20aa/users/5f5b9837682e492997c24009d9526e4d/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:40.076856 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05eff2cf-21d3-453b-9d9b-ceb8430158a7 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:40.082403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2756b6-344b-4c08-9920-5bef1a856c83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:40.083459 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e0c230-3dd7-4d30-bf66-f5c069ad5cf1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5575957500cd4f84b4d65520cece20aa, user_id=5f5b9837682e492997c24009d9526e4d, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:40.083605 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2756b6-344b-4c08-9920-5bef1a856c83 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:40.083990 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2756b6-344b-4c08-9920-5bef1a856c83 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:40.084362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2756b6-344b-4c08-9920-5bef1a856c83 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:40.085503 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2756b6-344b-4c08-9920-5bef1a856c83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:40.086222 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e0c230-3dd7-4d30-bf66-f5c069ad5cf1 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:40.088923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2756b6-344b-4c08-9920-5bef1a856c83 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:40.110440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 780/1575] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:40 2022] PUT /identity/v3/projects/5575957500cd4f84b4d65520cece20aa/users/5f5b9837682e492997c24009d9526e4d/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:40.110657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 796/1576] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:40 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:40.115521 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89dc7b82-bf44-4dc6-b7e0-f4f5d7649df3 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:40.115902 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89dc7b82-bf44-4dc6-b7e0-f4f5d7649df3 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:40.116263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89dc7b82-bf44-4dc6-b7e0-f4f5d7649df3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:40.120467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2756b6-344b-4c08-9920-5bef1a856c83 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:40.132740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2756b6-344b-4c08-9920-5bef1a856c83 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:40.156933 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-89dc7b82-bf44-4dc6-b7e0-f4f5d7649df3 None None] MFA Rules not processed for user `5f5b9837682e492997c24009d9526e4d`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:40.158942 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-89dc7b82-bf44-4dc6-b7e0-f4f5d7649df3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:40.206554 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0a3248-e3e6-45ee-9906-260a9969679b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:40.207285 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0a3248-e3e6-45ee-9906-260a9969679b admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:40.208009 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0a3248-e3e6-45ee-9906-260a9969679b admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:40.208387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0a3248-e3e6-45ee-9906-260a9969679b admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:40.209130 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0a3248-e3e6-45ee-9906-260a9969679b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:40.210843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0a3248-e3e6-45ee-9906-260a9969679b admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:40.230101 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 797/1577] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:40 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:40.230101 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0a3248-e3e6-45ee-9906-260a9969679b admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:40.230852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-9e0a3248-e3e6-45ee-9906-260a9969679b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:40.245174 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0a3248-e3e6-45ee-9906-260a9969679b admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:40.250661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720f28a7-a128-49b1-96f5-0dced7ae1167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:40.251515 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720f28a7-a128-49b1-96f5-0dced7ae1167 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:40.251890 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720f28a7-a128-49b1-96f5-0dced7ae1167 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:40.252198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720f28a7-a128-49b1-96f5-0dced7ae1167 admin admin] PATH_INFO: `/v3/projects/e9b4db8138434ee4ae5d9533c04f8b2a` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:40.253327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720f28a7-a128-49b1-96f5-0dced7ae1167 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e9b4db8138434ee4ae5d9533c04f8b2a)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:40.255258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720f28a7-a128-49b1-96f5-0dced7ae1167 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:40.263532 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 781/1578] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:40 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:57:40.264617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89dc7b82-bf44-4dc6-b7e0-f4f5d7649df3 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:40.275480 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89dc7b82-bf44-4dc6-b7e0-f4f5d7649df3 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:40.280804 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5d01be-4429-4b70-b343-50124a881889 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:40.281588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5d01be-4429-4b70-b343-50124a881889 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:40.281959 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5d01be-4429-4b70-b343-50124a881889 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:40.282333 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5d01be-4429-4b70-b343-50124a881889 admin admin] PATH_INFO: `/v3/projects/958ed455cb85472fa4dca0ab664bd977/users/84fa8bf189284bdca79c0f727f02affb/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:40.302978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5d01be-4429-4b70-b343-50124a881889 admin admin] RBAC: Authorizing `identity:create_grant(project_id=958ed455cb85472fa4dca0ab664bd977, user_id=84fa8bf189284bdca79c0f727f02affb, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:40.305704 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5d01be-4429-4b70-b343-50124a881889 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:40.314773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=92,keystone:INSERT=4,keystone:DELETE=4 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:40.316152 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-720f28a7-a128-49b1-96f5-0dced7ae1167 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e9b4db8138434ee4ae5d9533c04f8b2a', 'request_id': 'req-720f28a7-a128-49b1-96f5-0dced7ae1167'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:40.324226 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-720f28a7-a128-49b1-96f5-0dced7ae1167 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e9b4db8138434ee4ae5d9533c04f8b2a', 'request_id': 'req-720f28a7-a128-49b1-96f5-0dced7ae1167'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:40.324467 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 782/1579] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:40 2022] PUT /identity/v3/projects/958ed455cb85472fa4dca0ab664bd977/users/84fa8bf189284bdca79c0f727f02affb/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:40.325337 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5d01be-4429-4b70-b343-50124a881889 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:40.334212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-720f28a7-a128-49b1-96f5-0dced7ae1167 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e9b4db8138434ee4ae5d9533c04f8b2a', 'request_id': 'req-720f28a7-a128-49b1-96f5-0dced7ae1167'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:40.347788 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:40.350724 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-720f28a7-a128-49b1-96f5-0dced7ae1167 admin admin] The token cache is being invalidate because project e9b4db8138434ee4ae5d9533c04f8b2a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:40.351193 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-720f28a7-a128-49b1-96f5-0dced7ae1167 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:40.351937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-720f28a7-a128-49b1-96f5-0dced7ae1167 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:40.354985 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 798/1580] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:40 2022] DELETE /identity/v3/projects/e9b4db8138434ee4ae5d9533c04f8b2a => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:40.354985 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9806e1d4-80b5-48bc-b404-bec97b6ce118 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:40.355201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9806e1d4-80b5-48bc-b404-bec97b6ce118 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:40.355392 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9806e1d4-80b5-48bc-b404-bec97b6ce118 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:40.389719 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-9806e1d4-80b5-48bc-b404-bec97b6ce118 None None] MFA Rules not processed for user `84fa8bf189284bdca79c0f727f02affb`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:40.390981 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-9806e1d4-80b5-48bc-b404-bec97b6ce118 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:40.402542 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5d01be-4429-4b70-b343-50124a881889 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:40.408456 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2edeb2-94c3-4cbf-94be-b36a38a77eaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:40.409493 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2edeb2-94c3-4cbf-94be-b36a38a77eaa admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:40.410168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2edeb2-94c3-4cbf-94be-b36a38a77eaa admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:40.410447 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2edeb2-94c3-4cbf-94be-b36a38a77eaa admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:40.411323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2edeb2-94c3-4cbf-94be-b36a38a77eaa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:40.413412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2edeb2-94c3-4cbf-94be-b36a38a77eaa admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:40.451748 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 783/1581] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:40 2022] POST /identity/v3/projects => generated 385 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:40.454202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2edeb2-94c3-4cbf-94be-b36a38a77eaa admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:40.458724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-0a2edeb2-94c3-4cbf-94be-b36a38a77eaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:40.489706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 799/1582] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:40 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:40.490579 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9806e1d4-80b5-48bc-b404-bec97b6ce118 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:40.491917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-9806e1d4-80b5-48bc-b404-bec97b6ce118 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:40.516656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9806e1d4-80b5-48bc-b404-bec97b6ce118 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:40.559955 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2edeb2-94c3-4cbf-94be-b36a38a77eaa admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:40.576991 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8d8c76-6c3e-4e4d-9415-ec618764a530 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:40.577660 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8d8c76-6c3e-4e4d-9415-ec618764a530 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:40.577942 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8d8c76-6c3e-4e4d-9415-ec618764a530 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:40.578220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8d8c76-6c3e-4e4d-9415-ec618764a530 admin admin] PATH_INFO: `/v3/users/58eaf17b46004cdfbf66465f1775c605` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:40.585445 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8d8c76-6c3e-4e4d-9415-ec618764a530 admin admin] RBAC: Authorizing `identity:delete_user(user_id=58eaf17b46004cdfbf66465f1775c605)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:40.590692 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8d8c76-6c3e-4e4d-9415-ec618764a530 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:40.660498 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84f8e42-1112-46dd-bf59-e65dc466815d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:40.661606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84f8e42-1112-46dd-bf59-e65dc466815d admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:40.661985 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84f8e42-1112-46dd-bf59-e65dc466815d admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:40.662361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84f8e42-1112-46dd-bf59-e65dc466815d admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:40.663291 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84f8e42-1112-46dd-bf59-e65dc466815d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:40.666465 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84f8e42-1112-46dd-bf59-e65dc466815d admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:40.670734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8e8d8c76-6c3e-4e4d-9415-ec618764a530 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '58eaf17b46004cdfbf66465f1775c605'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:40.679789 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8e8d8c76-6c3e-4e4d-9415-ec618764a530 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '58eaf17b46004cdfbf66465f1775c605'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:40.681234 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8e8d8c76-6c3e-4e4d-9415-ec618764a530 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '58eaf17b46004cdfbf66465f1775c605'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:40.683092 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8e8d8c76-6c3e-4e4d-9415-ec618764a530 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '58eaf17b46004cdfbf66465f1775c605'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:40.698861 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 800/1583] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:40 2022] POST /identity/v3/users => generated 293 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:40.700396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84f8e42-1112-46dd-bf59-e65dc466815d admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:40.703334 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e84f8e42-1112-46dd-bf59-e65dc466815d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:40.703593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8e8d8c76-6c3e-4e4d-9415-ec618764a530 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '58eaf17b46004cdfbf66465f1775c605'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:40.720237 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84f8e42-1112-46dd-bf59-e65dc466815d admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:40.726709 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde47f3d-5887-4067-9b3a-03f639108de5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:40.727579 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde47f3d-5887-4067-9b3a-03f639108de5 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:40.727871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde47f3d-5887-4067-9b3a-03f639108de5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:40.728143 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde47f3d-5887-4067-9b3a-03f639108de5 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:40.729537 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde47f3d-5887-4067-9b3a-03f639108de5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:40.730976 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 784/1584] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:40 2022] DELETE /identity/v3/users/58eaf17b46004cdfbf66465f1775c605 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:40.731951 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8d8c76-6c3e-4e4d-9415-ec618764a530 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:40.734768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde47f3d-5887-4067-9b3a-03f639108de5 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:40.734927 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-8e8d8c76-6c3e-4e4d-9415-ec618764a530 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:40.765568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:40.802396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 801/1585] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:40 2022] POST /identity/v3/projects => generated 367 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:40.806216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde47f3d-5887-4067-9b3a-03f639108de5 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:40.834585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8d8c76-6c3e-4e4d-9415-ec618764a530 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:40.860349 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=92,keystone:DELETE=3,keystone:INSERT=5 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:40.901822 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde47f3d-5887-4067-9b3a-03f639108de5 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:40.914306 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab87b6eb-6bd8-4fc9-a768-3a9836418900 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:40.915495 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab87b6eb-6bd8-4fc9-a768-3a9836418900 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:40.916597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd495bc9-84e1-4d6f-af01-d22c2f035e84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:40.916868 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab87b6eb-6bd8-4fc9-a768-3a9836418900 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:40.917217 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd495bc9-84e1-4d6f-af01-d22c2f035e84 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:40.917648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd495bc9-84e1-4d6f-af01-d22c2f035e84 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:40.917753 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab87b6eb-6bd8-4fc9-a768-3a9836418900 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:40.917922 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd495bc9-84e1-4d6f-af01-d22c2f035e84 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:40.918633 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd495bc9-84e1-4d6f-af01-d22c2f035e84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:40.919067 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab87b6eb-6bd8-4fc9-a768-3a9836418900 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:40.920516 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd495bc9-84e1-4d6f-af01-d22c2f035e84 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:40.921837 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab87b6eb-6bd8-4fc9-a768-3a9836418900 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:40.935471 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 785/1586] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:40 2022] GET /identity/v3/roles => generated 1420 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:40.940455 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd495bc9-84e1-4d6f-af01-d22c2f035e84 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:40.953560 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd495bc9-84e1-4d6f-af01-d22c2f035e84 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:40.955961 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 802/1587] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:40 2022] POST /identity/v3/users => generated 290 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:57:40.960223 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab87b6eb-6bd8-4fc9-a768-3a9836418900 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:40.965977 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b628a71c-591d-4c7a-93a9-df8a65584ccf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:40.966709 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b628a71c-591d-4c7a-93a9-df8a65584ccf admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:40.967102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b628a71c-591d-4c7a-93a9-df8a65584ccf admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:40.967445 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b628a71c-591d-4c7a-93a9-df8a65584ccf admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:40.968191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b628a71c-591d-4c7a-93a9-df8a65584ccf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:40.970583 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b628a71c-591d-4c7a-93a9-df8a65584ccf admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:40.977961 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab87b6eb-6bd8-4fc9-a768-3a9836418900 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:40.986042 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11746c7d-2039-48b4-99c6-9a128262464b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:40.987076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11746c7d-2039-48b4-99c6-9a128262464b admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:40.988267 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11746c7d-2039-48b4-99c6-9a128262464b admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:40.988992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 786/1588] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:40 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:40.989407 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11746c7d-2039-48b4-99c6-9a128262464b admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:40.990699 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11746c7d-2039-48b4-99c6-9a128262464b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:40.992635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11746c7d-2039-48b4-99c6-9a128262464b admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:40.993590 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b628a71c-591d-4c7a-93a9-df8a65584ccf admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:41.007264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 803/1589] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:40 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:41.011683 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11746c7d-2039-48b4-99c6-9a128262464b admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:41.012009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b628a71c-591d-4c7a-93a9-df8a65584ccf admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:41.019885 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04876927-689d-463b-b68d-e524c0e3fc89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:41.020316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04876927-689d-463b-b68d-e524c0e3fc89 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:41.020665 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04876927-689d-463b-b68d-e524c0e3fc89 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:41.021013 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04876927-689d-463b-b68d-e524c0e3fc89 admin admin] PATH_INFO: `/v3/projects/32dd93585caa492f9d38a993db66a468/users/7f61ce7317814d2b941ca3f064cfda6f/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:41.027471 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11746c7d-2039-48b4-99c6-9a128262464b admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:41.035780 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bc7891-8e19-46c8-9e1b-14b624149fae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:41.036925 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bc7891-8e19-46c8-9e1b-14b624149fae admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:41.037635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bc7891-8e19-46c8-9e1b-14b624149fae admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:41.038673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bc7891-8e19-46c8-9e1b-14b624149fae admin admin] PATH_INFO: `/v3/projects/7f0698e33dab4d2abadd46023d4082e5/users/1700865366014acf9553a6b16148a0ac/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:41.046938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04876927-689d-463b-b68d-e524c0e3fc89 admin admin] RBAC: Authorizing `identity:create_grant(project_id=32dd93585caa492f9d38a993db66a468, user_id=7f61ce7317814d2b941ca3f064cfda6f, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:41.051540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04876927-689d-463b-b68d-e524c0e3fc89 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:41.060060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bc7891-8e19-46c8-9e1b-14b624149fae admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f0698e33dab4d2abadd46023d4082e5, user_id=1700865366014acf9553a6b16148a0ac, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:41.063202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bc7891-8e19-46c8-9e1b-14b624149fae admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:41.083799 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 787/1590] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:40 2022] PUT /identity/v3/projects/32dd93585caa492f9d38a993db66a468/users/7f61ce7317814d2b941ca3f064cfda6f/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:41.086247 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e6c75d-0345-4a0f-b4f1-4b0fa1036501 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:41.086740 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e6c75d-0345-4a0f-b4f1-4b0fa1036501 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:41.087221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e6c75d-0345-4a0f-b4f1-4b0fa1036501 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:41.091623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 804/1591] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:41 2022] PUT /identity/v3/projects/7f0698e33dab4d2abadd46023d4082e5/users/1700865366014acf9553a6b16148a0ac/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:41.093090 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bc7891-8e19-46c8-9e1b-14b624149fae admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:41.141113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-86e6c75d-0345-4a0f-b4f1-4b0fa1036501 None None] MFA Rules not processed for user `7f61ce7317814d2b941ca3f064cfda6f`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:41.143352 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-86e6c75d-0345-4a0f-b4f1-4b0fa1036501 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:41.174554 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bc7891-8e19-46c8-9e1b-14b624149fae admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:41.180623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aefa9574-7d05-4b6d-ae0f-8d381c21ec42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:41.183107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefa9574-7d05-4b6d-ae0f-8d381c21ec42 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:41.183107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefa9574-7d05-4b6d-ae0f-8d381c21ec42 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:41.183107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefa9574-7d05-4b6d-ae0f-8d381c21ec42 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:41.183107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aefa9574-7d05-4b6d-ae0f-8d381c21ec42 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:41.185181 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aefa9574-7d05-4b6d-ae0f-8d381c21ec42 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:41.200790 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 805/1592] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:41 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:41.201844 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aefa9574-7d05-4b6d-ae0f-8d381c21ec42 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:41.217866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aefa9574-7d05-4b6d-ae0f-8d381c21ec42 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:41.223423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9a919b-7c4c-48fa-98b1-9fe4b6f77e3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:41.224222 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9a919b-7c4c-48fa-98b1-9fe4b6f77e3d admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:41.224509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9a919b-7c4c-48fa-98b1-9fe4b6f77e3d admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:41.224793 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9a919b-7c4c-48fa-98b1-9fe4b6f77e3d admin admin] PATH_INFO: `/v3/projects/7f0698e33dab4d2abadd46023d4082e5/users/1700865366014acf9553a6b16148a0ac/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:41.227202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9a919b-7c4c-48fa-98b1-9fe4b6f77e3d admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f0698e33dab4d2abadd46023d4082e5, user_id=1700865366014acf9553a6b16148a0ac, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:41.231588 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9a919b-7c4c-48fa-98b1-9fe4b6f77e3d admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:41.265688 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 788/1593] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:41 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:41.280492 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e6c75d-0345-4a0f-b4f1-4b0fa1036501 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:41.287828 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 806/1594] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:41 2022] PUT /identity/v3/projects/7f0698e33dab4d2abadd46023d4082e5/users/1700865366014acf9553a6b16148a0ac/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:41.292318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822bcd1a-af28-4b65-be57-d2dd0539dc27 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:41.292473 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822bcd1a-af28-4b65-be57-d2dd0539dc27 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:41.292710 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822bcd1a-af28-4b65-be57-d2dd0539dc27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:41.319166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e6c75d-0345-4a0f-b4f1-4b0fa1036501 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:41.349966 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-822bcd1a-af28-4b65-be57-d2dd0539dc27 None None] MFA Rules not processed for user `1700865366014acf9553a6b16148a0ac`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:41.351751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-822bcd1a-af28-4b65-be57-d2dd0539dc27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:41.352318 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:41.434382 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6463622f-aeef-4712-a65a-2d9be81ff17a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:41.441635 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6463622f-aeef-4712-a65a-2d9be81ff17a admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:41.442219 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6463622f-aeef-4712-a65a-2d9be81ff17a admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:41.443678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6463622f-aeef-4712-a65a-2d9be81ff17a admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:41.444759 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6463622f-aeef-4712-a65a-2d9be81ff17a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:41.446804 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6463622f-aeef-4712-a65a-2d9be81ff17a admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:41.463723 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:41.487402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 807/1595] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:41 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:41.492316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 789/1596] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:41 2022] POST /identity/v3/projects => generated 385 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:41.496392 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822bcd1a-af28-4b65-be57-d2dd0539dc27 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:41.501836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6463622f-aeef-4712-a65a-2d9be81ff17a admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:41.505515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6463622f-aeef-4712-a65a-2d9be81ff17a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:41.516434 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822bcd1a-af28-4b65-be57-d2dd0539dc27 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:41.601708 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6463622f-aeef-4712-a65a-2d9be81ff17a admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:41.610008 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34397036-4f0e-4a1f-8684-ad5df0710f38 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:41.610777 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34397036-4f0e-4a1f-8684-ad5df0710f38 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:41.611043 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34397036-4f0e-4a1f-8684-ad5df0710f38 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:41.611413 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34397036-4f0e-4a1f-8684-ad5df0710f38 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:41.626732 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6947f52-90fa-44ea-bec4-c511b1e663a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:41.627347 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6947f52-90fa-44ea-bec4-c511b1e663a9 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:41.627652 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6947f52-90fa-44ea-bec4-c511b1e663a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:41.627925 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6947f52-90fa-44ea-bec4-c511b1e663a9 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:41.628567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6947f52-90fa-44ea-bec4-c511b1e663a9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:41.630621 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6947f52-90fa-44ea-bec4-c511b1e663a9 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:41.659891 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 808/1597] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:41 2022] POST /identity/v3/users => generated 299 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:41.669564 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6947f52-90fa-44ea-bec4-c511b1e663a9 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:41.689692 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6947f52-90fa-44ea-bec4-c511b1e663a9 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:41.695750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720f7a7c-964e-4f0f-8667-b16709c1ec80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:41.696394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720f7a7c-964e-4f0f-8667-b16709c1ec80 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:41.696671 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720f7a7c-964e-4f0f-8667-b16709c1ec80 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:41.697695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720f7a7c-964e-4f0f-8667-b16709c1ec80 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:41.698352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720f7a7c-964e-4f0f-8667-b16709c1ec80 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:41.700282 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720f7a7c-964e-4f0f-8667-b16709c1ec80 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:41.705714 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34397036-4f0e-4a1f-8684-ad5df0710f38 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:41.713152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34397036-4f0e-4a1f-8684-ad5df0710f38 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:41.715739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 809/1598] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:41 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:41.719035 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720f7a7c-964e-4f0f-8667-b16709c1ec80 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:41.726444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 790/1599] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:41 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:41.732069 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720f7a7c-964e-4f0f-8667-b16709c1ec80 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:41.737298 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339514f3-1502-47f5-845c-5ce9385c5598 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:41.738047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339514f3-1502-47f5-845c-5ce9385c5598 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:41.738429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339514f3-1502-47f5-845c-5ce9385c5598 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:41.738772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339514f3-1502-47f5-845c-5ce9385c5598 admin admin] PATH_INFO: `/v3/projects/aaf529402e634ed89dc6d4fbf8b0696a/users/03c825567068473eb8bbdcdcf3f70861/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:41.755121 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339514f3-1502-47f5-845c-5ce9385c5598 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aaf529402e634ed89dc6d4fbf8b0696a, user_id=03c825567068473eb8bbdcdcf3f70861, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:41.757958 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339514f3-1502-47f5-845c-5ce9385c5598 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:41.781081 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 810/1600] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:41 2022] PUT /identity/v3/projects/aaf529402e634ed89dc6d4fbf8b0696a/users/03c825567068473eb8bbdcdcf3f70861/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:41.783102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34397036-4f0e-4a1f-8684-ad5df0710f38 service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:41.796918 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34397036-4f0e-4a1f-8684-ad5df0710f38 service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:41.860627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:41.873503 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76914485-c0ee-4cca-8339-f3608c925e1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:41.874857 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76914485-c0ee-4cca-8339-f3608c925e1c admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:41.875232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76914485-c0ee-4cca-8339-f3608c925e1c admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:41.875650 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76914485-c0ee-4cca-8339-f3608c925e1c admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:41.876523 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76914485-c0ee-4cca-8339-f3608c925e1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:41.878597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76914485-c0ee-4cca-8339-f3608c925e1c admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:41.897259 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 791/1601] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:41 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:41.902588 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339514f3-1502-47f5-845c-5ce9385c5598 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:41.917036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339514f3-1502-47f5-845c-5ce9385c5598 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:41.924705 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42e2554-7fec-4b14-9e55-ed7404853327 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:41.925719 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42e2554-7fec-4b14-9e55-ed7404853327 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:41.926317 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42e2554-7fec-4b14-9e55-ed7404853327 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:41.926806 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42e2554-7fec-4b14-9e55-ed7404853327 admin admin] PATH_INFO: `/v3/projects/aaf529402e634ed89dc6d4fbf8b0696a/users/03c825567068473eb8bbdcdcf3f70861/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:41.930759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42e2554-7fec-4b14-9e55-ed7404853327 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aaf529402e634ed89dc6d4fbf8b0696a, user_id=03c825567068473eb8bbdcdcf3f70861, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:41.937910 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42e2554-7fec-4b14-9e55-ed7404853327 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:41.971960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 811/1602] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:41 2022] PUT /identity/v3/projects/aaf529402e634ed89dc6d4fbf8b0696a/users/03c825567068473eb8bbdcdcf3f70861/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:41.978626 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fca900-d769-4ed5-bdf0-ed6a67e7e99d None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:41.978626 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fca900-d769-4ed5-bdf0-ed6a67e7e99d None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:41.978626 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fca900-d769-4ed5-bdf0-ed6a67e7e99d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:42.079439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-73fca900-d769-4ed5-bdf0-ed6a67e7e99d None None] MFA Rules not processed for user `03c825567068473eb8bbdcdcf3f70861`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:42.080959 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-73fca900-d769-4ed5-bdf0-ed6a67e7e99d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:42.102029 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42e2554-7fec-4b14-9e55-ed7404853327 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:42.104866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a42e2554-7fec-4b14-9e55-ed7404853327 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:42.256947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42e2554-7fec-4b14-9e55-ed7404853327 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:42.271226 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93ef887-efb5-4165-9f84-909727d6c214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:42.272863 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93ef887-efb5-4165-9f84-909727d6c214 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:42.273149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93ef887-efb5-4165-9f84-909727d6c214 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:42.273425 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93ef887-efb5-4165-9f84-909727d6c214 admin admin] PATH_INFO: `/v3/projects/e796b1f09e7b470fb251ac2c88abd264` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:42.283528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 792/1603] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:41 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:42.285334 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93ef887-efb5-4165-9f84-909727d6c214 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e796b1f09e7b470fb251ac2c88abd264)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:42.286478 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fca900-d769-4ed5-bdf0-ed6a67e7e99d None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:42.298554 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93ef887-efb5-4165-9f84-909727d6c214 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:42.405322 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:42.409065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fca900-d769-4ed5-bdf0-ed6a67e7e99d None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:42.423642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b93ef887-efb5-4165-9f84-909727d6c214 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e796b1f09e7b470fb251ac2c88abd264', 'request_id': 'req-b93ef887-efb5-4165-9f84-909727d6c214'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:42.441698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b93ef887-efb5-4165-9f84-909727d6c214 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e796b1f09e7b470fb251ac2c88abd264', 'request_id': 'req-b93ef887-efb5-4165-9f84-909727d6c214'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:42.458329 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b93ef887-efb5-4165-9f84-909727d6c214 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e796b1f09e7b470fb251ac2c88abd264', 'request_id': 'req-b93ef887-efb5-4165-9f84-909727d6c214'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:42.495615 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b13979a-9cf0-4fcb-9b72-a4f5fcd02a97 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:42.496256 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b13979a-9cf0-4fcb-9b72-a4f5fcd02a97 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:42.496582 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b13979a-9cf0-4fcb-9b72-a4f5fcd02a97 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:42.496863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b13979a-9cf0-4fcb-9b72-a4f5fcd02a97 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:42.498781 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6b13979a-9cf0-4fcb-9b72-a4f5fcd02a97 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:42.514969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b93ef887-efb5-4165-9f84-909727d6c214 admin admin] The token cache is being invalidate because project e796b1f09e7b470fb251ac2c88abd264 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:42.515394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b93ef887-efb5-4165-9f84-909727d6c214 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:42.517279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b93ef887-efb5-4165-9f84-909727d6c214 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:42.523373 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 812/1604] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:42 2022] DELETE /identity/v3/projects/e796b1f09e7b470fb251ac2c88abd264 => generated 0 bytes in 422 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:42.529496 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81c8fed-90a4-45bd-b825-5a16cd21e104 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:42.530228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81c8fed-90a4-45bd-b825-5a16cd21e104 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:42.530509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81c8fed-90a4-45bd-b825-5a16cd21e104 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:42.616476 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:42.632083 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-e81c8fed-90a4-45bd-b825-5a16cd21e104 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:42.637149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e81c8fed-90a4-45bd-b825-5a16cd21e104 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:42.646569 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b13979a-9cf0-4fcb-9b72-a4f5fcd02a97 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:42.652373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b13979a-9cf0-4fcb-9b72-a4f5fcd02a97 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:42.704451 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 793/1605] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:42 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:42.816124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 813/1606] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:42.824420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21258bd4-66be-4b06-b98d-450bbf9249fe None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:42.824760 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21258bd4-66be-4b06-b98d-450bbf9249fe None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:42.825041 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21258bd4-66be-4b06-b98d-450bbf9249fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:42.854088 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-21258bd4-66be-4b06-b98d-450bbf9249fe None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:42.855816 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-21258bd4-66be-4b06-b98d-450bbf9249fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:42.871313 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 794/1607] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:42.872324 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81c8fed-90a4-45bd-b825-5a16cd21e104 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:42.886508 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81c8fed-90a4-45bd-b825-5a16cd21e104 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:42.892759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dd9478-a4ea-4591-9608-979e7f9962cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:42.893326 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dd9478-a4ea-4591-9608-979e7f9962cb admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:42.893607 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dd9478-a4ea-4591-9608-979e7f9962cb admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:42.893876 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dd9478-a4ea-4591-9608-979e7f9962cb admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:42.894526 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dd9478-a4ea-4591-9608-979e7f9962cb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:57:42.894839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dd9478-a4ea-4591-9608-979e7f9962cb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:42.897554 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dd9478-a4ea-4591-9608-979e7f9962cb admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:42.915993 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 814/1608] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:57:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:57:42.921093 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21258bd4-66be-4b06-b98d-450bbf9249fe None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:42.932605 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21258bd4-66be-4b06-b98d-450bbf9249fe None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:42.951289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2d3e57-ba44-4c29-a8cc-4919de31322e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:42.952117 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d3e57-ba44-4c29-a8cc-4919de31322e admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:42.952399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d3e57-ba44-4c29-a8cc-4919de31322e admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:42.952751 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d3e57-ba44-4c29-a8cc-4919de31322e admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:42.955869 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2d3e57-ba44-4c29-a8cc-4919de31322e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:42.961835 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2d3e57-ba44-4c29-a8cc-4919de31322e admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:43.035100 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 795/1609] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:42 2022] POST /identity/v3/projects => generated 365 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:43.037269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dd9478-a4ea-4591-9608-979e7f9962cb admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:43.061590 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dd9478-a4ea-4591-9608-979e7f9962cb admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:43.264541 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0f1f43-3051-4735-a74d-fb9c25972a50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:43.274599 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0f1f43-3051-4735-a74d-fb9c25972a50 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:43.274599 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0f1f43-3051-4735-a74d-fb9c25972a50 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:43.274599 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0f1f43-3051-4735-a74d-fb9c25972a50 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:43.275403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0f1f43-3051-4735-a74d-fb9c25972a50 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:43.280674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0f1f43-3051-4735-a74d-fb9c25972a50 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:43.339586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 815/1610] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:43 2022] POST /identity/v3/users => generated 283 bytes in 293 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:43.341531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2d3e57-ba44-4c29-a8cc-4919de31322e admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:43.358429 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2d3e57-ba44-4c29-a8cc-4919de31322e admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:43.372614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e88b904-6728-4152-8281-10273f14bb5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:43.373893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e88b904-6728-4152-8281-10273f14bb5e admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:43.374370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e88b904-6728-4152-8281-10273f14bb5e admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:43.375020 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e88b904-6728-4152-8281-10273f14bb5e admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:43.375873 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e88b904-6728-4152-8281-10273f14bb5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:43.378150 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e88b904-6728-4152-8281-10273f14bb5e admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:43.402916 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 796/1611] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:43 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:43.407131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0f1f43-3051-4735-a74d-fb9c25972a50 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:43.418161 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0f1f43-3051-4735-a74d-fb9c25972a50 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:43.438725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f294754-1a17-42fa-9593-d62b64fc5d4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:43.439357 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f294754-1a17-42fa-9593-d62b64fc5d4d admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:43.439690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f294754-1a17-42fa-9593-d62b64fc5d4d admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:43.441198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f294754-1a17-42fa-9593-d62b64fc5d4d admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:43.443142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f294754-1a17-42fa-9593-d62b64fc5d4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:43.449294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f294754-1a17-42fa-9593-d62b64fc5d4d admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:43.484398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 816/1612] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:43 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:43.489538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e88b904-6728-4152-8281-10273f14bb5e admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:43.516122 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e88b904-6728-4152-8281-10273f14bb5e admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:43.532746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1688f9ee-7613-45c5-8c6b-0d6c0f35671e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:43.535149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1688f9ee-7613-45c5-8c6b-0d6c0f35671e admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:43.535899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1688f9ee-7613-45c5-8c6b-0d6c0f35671e admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:43.538137 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1688f9ee-7613-45c5-8c6b-0d6c0f35671e admin admin] PATH_INFO: `/v3/projects/cf9ce4b4c913434ab045fd477358afc2/users/d16822ed88ee49338a24f0ad3aca4f7a/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:43.587499 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1688f9ee-7613-45c5-8c6b-0d6c0f35671e admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf9ce4b4c913434ab045fd477358afc2, user_id=d16822ed88ee49338a24f0ad3aca4f7a, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:43.596100 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1688f9ee-7613-45c5-8c6b-0d6c0f35671e admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:43.620628 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 797/1613] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:43 2022] PUT /identity/v3/projects/cf9ce4b4c913434ab045fd477358afc2/users/d16822ed88ee49338a24f0ad3aca4f7a/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:43.630907 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d95489-d619-43b5-95e9-5760ffce097f None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:43.631215 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d95489-d619-43b5-95e9-5760ffce097f None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:43.631488 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d95489-d619-43b5-95e9-5760ffce097f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:43.700340 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-72d95489-d619-43b5-95e9-5760ffce097f None None] MFA Rules not processed for user `d16822ed88ee49338a24f0ad3aca4f7a`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:43.701697 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-72d95489-d619-43b5-95e9-5760ffce097f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:43.750893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:43.816352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 817/1614] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:43 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:43.821617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1688f9ee-7613-45c5-8c6b-0d6c0f35671e admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:43.910083 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1688f9ee-7613-45c5-8c6b-0d6c0f35671e admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:43.917073 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd36d1c7-a9b2-448a-8ed4-a213ed177c38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:43.917700 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd36d1c7-a9b2-448a-8ed4-a213ed177c38 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:43.918103 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd36d1c7-a9b2-448a-8ed4-a213ed177c38 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:43.918588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd36d1c7-a9b2-448a-8ed4-a213ed177c38 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:43.919355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd36d1c7-a9b2-448a-8ed4-a213ed177c38 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:43.921950 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd36d1c7-a9b2-448a-8ed4-a213ed177c38 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:43.979807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 798/1615] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:43 2022] POST /identity/v3/projects => generated 367 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:57:43.981431 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d95489-d619-43b5-95e9-5760ffce097f None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:43.997546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d95489-d619-43b5-95e9-5760ffce097f None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:44.085289 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4872b22e-4663-4537-bbc7-8357da8835d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:44.086191 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4872b22e-4663-4537-bbc7-8357da8835d9 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:44.086541 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4872b22e-4663-4537-bbc7-8357da8835d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:44.086884 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4872b22e-4663-4537-bbc7-8357da8835d9 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:44.087990 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4872b22e-4663-4537-bbc7-8357da8835d9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:44.090080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4872b22e-4663-4537-bbc7-8357da8835d9 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:44.119595 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 818/1616] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:43 2022] POST /identity/v3/users => generated 290 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:44.122646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd36d1c7-a9b2-448a-8ed4-a213ed177c38 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:44.135424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd36d1c7-a9b2-448a-8ed4-a213ed177c38 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:44.141977 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c07843-1346-4a5c-b763-373e0369d1a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:44.143091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c07843-1346-4a5c-b763-373e0369d1a6 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:44.143591 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c07843-1346-4a5c-b763-373e0369d1a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:44.144250 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c07843-1346-4a5c-b763-373e0369d1a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:44.145085 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c07843-1346-4a5c-b763-373e0369d1a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:44.147018 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c07843-1346-4a5c-b763-373e0369d1a6 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:44.162697 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 799/1617] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:44 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:44.166713 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4872b22e-4663-4537-bbc7-8357da8835d9 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:44.178039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4872b22e-4663-4537-bbc7-8357da8835d9 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:44.208693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816c13a8-d76e-4e66-b247-37202c39e9b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:44.209296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816c13a8-d76e-4e66-b247-37202c39e9b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:44.209580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816c13a8-d76e-4e66-b247-37202c39e9b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:44.209869 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816c13a8-d76e-4e66-b247-37202c39e9b6 admin admin] PATH_INFO: `/v3/projects/b6b3250de59043739c02fb6839b0be73/users/be65a1124e934b25b277144dc05c2099/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:44.233372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816c13a8-d76e-4e66-b247-37202c39e9b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6b3250de59043739c02fb6839b0be73, user_id=be65a1124e934b25b277144dc05c2099, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:44.236421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816c13a8-d76e-4e66-b247-37202c39e9b6 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:44.264921 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 819/1618] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:44 2022] PUT /identity/v3/projects/b6b3250de59043739c02fb6839b0be73/users/be65a1124e934b25b277144dc05c2099/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:44.270416 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c07843-1346-4a5c-b763-373e0369d1a6 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:44.294207 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c07843-1346-4a5c-b763-373e0369d1a6 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:44.453379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:44.470278 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc5e532-afe5-4df1-ac1a-c78e9ae6b687 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:44.471139 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc5e532-afe5-4df1-ac1a-c78e9ae6b687 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:44.472519 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc5e532-afe5-4df1-ac1a-c78e9ae6b687 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:44.472966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc5e532-afe5-4df1-ac1a-c78e9ae6b687 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:44.473764 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc5e532-afe5-4df1-ac1a-c78e9ae6b687 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:44.477170 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc5e532-afe5-4df1-ac1a-c78e9ae6b687 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:44.500045 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 800/1619] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:44 2022] GET /identity/v3/roles => generated 1420 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:44.504256 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816c13a8-d76e-4e66-b247-37202c39e9b6 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:44.533522 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816c13a8-d76e-4e66-b247-37202c39e9b6 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:44.541524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edafb33-23e7-4dd6-a3ce-477ddf0042de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:44.542314 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edafb33-23e7-4dd6-a3ce-477ddf0042de admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:44.542692 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edafb33-23e7-4dd6-a3ce-477ddf0042de admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:44.543352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edafb33-23e7-4dd6-a3ce-477ddf0042de admin admin] PATH_INFO: `/v3/projects/b6b3250de59043739c02fb6839b0be73/users/be65a1124e934b25b277144dc05c2099/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:44.548711 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edafb33-23e7-4dd6-a3ce-477ddf0042de admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6b3250de59043739c02fb6839b0be73, user_id=be65a1124e934b25b277144dc05c2099, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:44.552405 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edafb33-23e7-4dd6-a3ce-477ddf0042de admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:44.563601 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc5e532-afe5-4df1-ac1a-c78e9ae6b687 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:44.565448 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-bbc5e532-afe5-4df1-ac1a-c78e9ae6b687 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:44.583212 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc5e532-afe5-4df1-ac1a-c78e9ae6b687 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:44.599609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 820/1620] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:44 2022] PUT /identity/v3/projects/b6b3250de59043739c02fb6839b0be73/users/be65a1124e934b25b277144dc05c2099/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:44.604862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e78ea95-3a73-49d2-9a8a-4af51cd01cef None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:44.605375 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e78ea95-3a73-49d2-9a8a-4af51cd01cef None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:44.605745 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e78ea95-3a73-49d2-9a8a-4af51cd01cef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:44.674454 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-7e78ea95-3a73-49d2-9a8a-4af51cd01cef None None] MFA Rules not processed for user `be65a1124e934b25b277144dc05c2099`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:44.675762 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7e78ea95-3a73-49d2-9a8a-4af51cd01cef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:44.709473 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2667ea-7994-4baf-b0cd-527ade8ec137 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:44.710235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2667ea-7994-4baf-b0cd-527ade8ec137 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:44.710609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2667ea-7994-4baf-b0cd-527ade8ec137 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:44.711036 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2667ea-7994-4baf-b0cd-527ade8ec137 admin admin] PATH_INFO: `/v3/users/84fa8bf189284bdca79c0f727f02affb` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:44.712509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e2667ea-7994-4baf-b0cd-527ade8ec137 admin admin] RBAC: Authorizing `identity:delete_user(user_id=84fa8bf189284bdca79c0f727f02affb)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:44.714898 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e2667ea-7994-4baf-b0cd-527ade8ec137 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:44.804199 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:44.809766 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 821/1621] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:44 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:57:44.814320 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e78ea95-3a73-49d2-9a8a-4af51cd01cef None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:44.823030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5e2667ea-7994-4baf-b0cd-527ade8ec137 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '84fa8bf189284bdca79c0f727f02affb'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:44.823384 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7e78ea95-3a73-49d2-9a8a-4af51cd01cef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:44.833328 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5e2667ea-7994-4baf-b0cd-527ade8ec137 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '84fa8bf189284bdca79c0f727f02affb'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:44.833524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5e2667ea-7994-4baf-b0cd-527ade8ec137 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '84fa8bf189284bdca79c0f727f02affb'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:44.834253 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5e2667ea-7994-4baf-b0cd-527ade8ec137 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '84fa8bf189284bdca79c0f727f02affb'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:44.852647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5e2667ea-7994-4baf-b0cd-527ade8ec137 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '84fa8bf189284bdca79c0f727f02affb'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:44.874908 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 801/1622] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:44 2022] DELETE /identity/v3/users/84fa8bf189284bdca79c0f727f02affb => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:44.875850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2667ea-7994-4baf-b0cd-527ade8ec137 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:44.877937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5e2667ea-7994-4baf-b0cd-527ade8ec137 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:44.917451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e78ea95-3a73-49d2-9a8a-4af51cd01cef None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:44.987130 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2667ea-7994-4baf-b0cd-527ade8ec137 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:45.000637 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143845d9-d550-45ff-adae-67cd2770acf1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:45.001280 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143845d9-d550-45ff-adae-67cd2770acf1 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:45.001715 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143845d9-d550-45ff-adae-67cd2770acf1 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:45.001999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143845d9-d550-45ff-adae-67cd2770acf1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:45.003858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-143845d9-d550-45ff-adae-67cd2770acf1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:45.077001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:45.086185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00808c91-8b76-4193-af55-2e15239f8230 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:45.087443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00808c91-8b76-4193-af55-2e15239f8230 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:45.087776 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00808c91-8b76-4193-af55-2e15239f8230 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:45.088458 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00808c91-8b76-4193-af55-2e15239f8230 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:45.100273 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-00808c91-8b76-4193-af55-2e15239f8230 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:45.143425 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143845d9-d550-45ff-adae-67cd2770acf1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:45.145178 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143845d9-d550-45ff-adae-67cd2770acf1 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:45.157727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 802/1623] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:44 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 282 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:45.159433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211649dc-13a2-44ee-94e9-69e713835297 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:45.159944 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211649dc-13a2-44ee-94e9-69e713835297 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:45.160371 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211649dc-13a2-44ee-94e9-69e713835297 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:45.185900 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-211649dc-13a2-44ee-94e9-69e713835297 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:45.187910 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-211649dc-13a2-44ee-94e9-69e713835297 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:45.278608 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00808c91-8b76-4193-af55-2e15239f8230 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:45.280394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00808c91-8b76-4193-af55-2e15239f8230 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:45.304891 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 822/1624] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:44 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 492 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:45.305648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 803/1625] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:45.330893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00808c91-8b76-4193-af55-2e15239f8230 service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:45.333658 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-00808c91-8b76-4193-af55-2e15239f8230 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:45.427436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211649dc-13a2-44ee-94e9-69e713835297 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:45.449107 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211649dc-13a2-44ee-94e9-69e713835297 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:45.460516 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e17ca2f-cb49-4731-aae7-1f4369cc93d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:45.461992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e17ca2f-cb49-4731-aae7-1f4369cc93d5 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:45.462480 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e17ca2f-cb49-4731-aae7-1f4369cc93d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:45.463619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e17ca2f-cb49-4731-aae7-1f4369cc93d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:45.465767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-3e17ca2f-cb49-4731-aae7-1f4369cc93d5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:45.467706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00808c91-8b76-4193-af55-2e15239f8230 service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:45.494279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458598b5-300e-4746-a20b-f9ee206c80fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:45.494843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458598b5-300e-4746-a20b-f9ee206c80fa service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:45.495126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458598b5-300e-4746-a20b-f9ee206c80fa service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:45.495397 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458598b5-300e-4746-a20b-f9ee206c80fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:45.552690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458598b5-300e-4746-a20b-f9ee206c80fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:45.559107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458598b5-300e-4746-a20b-f9ee206c80fa service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:45.589426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 823/1626] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:45.622209 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e17ca2f-cb49-4731-aae7-1f4369cc93d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:45.624818 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e17ca2f-cb49-4731-aae7-1f4369cc93d5 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:45.651329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 804/1627] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:45 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:57:46.334846 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458598b5-300e-4746-a20b-f9ee206c80fa service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:46.338705 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-458598b5-300e-4746-a20b-f9ee206c80fa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:46.366071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e17ca2f-cb49-4731-aae7-1f4369cc93d5 service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:46.367292 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458598b5-300e-4746-a20b-f9ee206c80fa service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:46.376872 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e17ca2f-cb49-4731-aae7-1f4369cc93d5 service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:46.388234 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f6af3c-ec66-4c40-9eda-b488bdc2bb93 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:46.388936 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f6af3c-ec66-4c40-9eda-b488bdc2bb93 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:46.389224 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f6af3c-ec66-4c40-9eda-b488bdc2bb93 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:46.389504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f6af3c-ec66-4c40-9eda-b488bdc2bb93 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:46.395173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-51f6af3c-ec66-4c40-9eda-b488bdc2bb93 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:46.402796 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:46.406753 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:46.412158 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:46.412457 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40 admin admin] PATH_INFO: `/v3/projects/958ed455cb85472fa4dca0ab664bd977` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:46.419756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40 admin admin] RBAC: Authorizing `identity:delete_project(project_id=958ed455cb85472fa4dca0ab664bd977)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:46.421945 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:46.468805 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:46.512510 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:46.517820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f6af3c-ec66-4c40-9eda-b488bdc2bb93 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:46.523713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f6af3c-ec66-4c40-9eda-b488bdc2bb93 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:46.546847 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 805/1628] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:46 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:46.553950 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '958ed455cb85472fa4dca0ab664bd977', 'request_id': 'req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:46.563232 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '958ed455cb85472fa4dca0ab664bd977', 'request_id': 'req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:46.583414 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '958ed455cb85472fa4dca0ab664bd977', 'request_id': 'req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:46.606131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40 admin admin] The token cache is being invalidate because project 958ed455cb85472fa4dca0ab664bd977 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:46.606434 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:46.607148 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:46.614510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 824/1629] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:46 2022] DELETE /identity/v3/projects/958ed455cb85472fa4dca0ab664bd977 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:46.618621 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f6af3c-ec66-4c40-9eda-b488bdc2bb93 service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:46.635806 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f6af3c-ec66-4c40-9eda-b488bdc2bb93 service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:46.732337 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876cc184-8dbe-4a8b-97e5-8555f5fe0407 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:46.732981 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876cc184-8dbe-4a8b-97e5-8555f5fe0407 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:46.733363 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876cc184-8dbe-4a8b-97e5-8555f5fe0407 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:46.733521 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876cc184-8dbe-4a8b-97e5-8555f5fe0407 admin admin] PATH_INFO: `/v3/users/1700865366014acf9553a6b16148a0ac` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:46.735076 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876cc184-8dbe-4a8b-97e5-8555f5fe0407 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1700865366014acf9553a6b16148a0ac)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:46.737201 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876cc184-8dbe-4a8b-97e5-8555f5fe0407 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:46.793178 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-876cc184-8dbe-4a8b-97e5-8555f5fe0407 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1700865366014acf9553a6b16148a0ac'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:46.803485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-876cc184-8dbe-4a8b-97e5-8555f5fe0407 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1700865366014acf9553a6b16148a0ac'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:46.804305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-876cc184-8dbe-4a8b-97e5-8555f5fe0407 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1700865366014acf9553a6b16148a0ac'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:46.805011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-876cc184-8dbe-4a8b-97e5-8555f5fe0407 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1700865366014acf9553a6b16148a0ac'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:46.817704 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-876cc184-8dbe-4a8b-97e5-8555f5fe0407 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1700865366014acf9553a6b16148a0ac'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:46.832198 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 806/1630] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:46 2022] DELETE /identity/v3/users/1700865366014acf9553a6b16148a0ac => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:47.154513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:47.156032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:47.169685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37f6099-1b86-44c9-a89b-e69e9e8d3e40 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:47.226892 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fd4ab0-79f5-47bb-adff-ba1c44dae1f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:47.227525 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fd4ab0-79f5-47bb-adff-ba1c44dae1f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:47.228029 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fd4ab0-79f5-47bb-adff-ba1c44dae1f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:47.228304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fd4ab0-79f5-47bb-adff-ba1c44dae1f2 admin admin] PATH_INFO: `/v3/projects/7f0698e33dab4d2abadd46023d4082e5` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:47.229412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fd4ab0-79f5-47bb-adff-ba1c44dae1f2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f0698e33dab4d2abadd46023d4082e5)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:47.231308 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fd4ab0-79f5-47bb-adff-ba1c44dae1f2 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:47.277001 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-36fd4ab0-79f5-47bb-adff-ba1c44dae1f2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f0698e33dab4d2abadd46023d4082e5', 'request_id': 'req-36fd4ab0-79f5-47bb-adff-ba1c44dae1f2'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:47.283268 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-36fd4ab0-79f5-47bb-adff-ba1c44dae1f2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f0698e33dab4d2abadd46023d4082e5', 'request_id': 'req-36fd4ab0-79f5-47bb-adff-ba1c44dae1f2'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:47.289803 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-36fd4ab0-79f5-47bb-adff-ba1c44dae1f2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f0698e33dab4d2abadd46023d4082e5', 'request_id': 'req-36fd4ab0-79f5-47bb-adff-ba1c44dae1f2'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:47.301635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-36fd4ab0-79f5-47bb-adff-ba1c44dae1f2 admin admin] The token cache is being invalidate because project 7f0698e33dab4d2abadd46023d4082e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:47.301924 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-36fd4ab0-79f5-47bb-adff-ba1c44dae1f2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:47.302538 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-36fd4ab0-79f5-47bb-adff-ba1c44dae1f2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:47.304064 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 825/1631] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:47 2022] DELETE /identity/v3/projects/7f0698e33dab4d2abadd46023d4082e5 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:47.310451 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db507ec6-09d4-4345-9c57-e07b10479bee None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:47.310742 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db507ec6-09d4-4345-9c57-e07b10479bee None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:47.311022 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db507ec6-09d4-4345-9c57-e07b10479bee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:47.331759 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-db507ec6-09d4-4345-9c57-e07b10479bee None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:47.333236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-db507ec6-09d4-4345-9c57-e07b10479bee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:47.397459 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 807/1632] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:47.406829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f0a357-5847-4804-a96f-5bf4658ac1f8 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:47.407115 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f0a357-5847-4804-a96f-5bf4658ac1f8 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:47.407394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f0a357-5847-4804-a96f-5bf4658ac1f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:47.428428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-83f0a357-5847-4804-a96f-5bf4658ac1f8 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:47.429631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-83f0a357-5847-4804-a96f-5bf4658ac1f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:47.438333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 826/1633] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:47.442981 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db507ec6-09d4-4345-9c57-e07b10479bee None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:47.453276 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db507ec6-09d4-4345-9c57-e07b10479bee None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:47.460141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a265ca7-1517-4a90-9ac2-72b265b72242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:47.460754 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a265ca7-1517-4a90-9ac2-72b265b72242 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:47.461037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a265ca7-1517-4a90-9ac2-72b265b72242 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:47.461315 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a265ca7-1517-4a90-9ac2-72b265b72242 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:47.461959 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a265ca7-1517-4a90-9ac2-72b265b72242 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:57:47.462263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a265ca7-1517-4a90-9ac2-72b265b72242 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:47.463890 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a265ca7-1517-4a90-9ac2-72b265b72242 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:47.475717 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:47.480911 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 808/1634] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:57:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:57:47.484967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f0a357-5847-4804-a96f-5bf4658ac1f8 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:47.497167 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f0a357-5847-4804-a96f-5bf4658ac1f8 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:47.503132 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e35b660-f8e1-4e9e-8a41-057501a00f2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:47.504056 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e35b660-f8e1-4e9e-8a41-057501a00f2d admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:47.504498 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e35b660-f8e1-4e9e-8a41-057501a00f2d admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:47.504770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e35b660-f8e1-4e9e-8a41-057501a00f2d admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:47.505572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e35b660-f8e1-4e9e-8a41-057501a00f2d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:47.507663 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e35b660-f8e1-4e9e-8a41-057501a00f2d admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:47.538992 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:47.552613 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 827/1635] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:47 2022] POST /identity/v3/projects => generated 373 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:47.554179 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a265ca7-1517-4a90-9ac2-72b265b72242 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:47.571139 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a265ca7-1517-4a90-9ac2-72b265b72242 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:47.645029 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3bf576f-fef8-453b-be55-039d02372b3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:47.645662 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bf576f-fef8-453b-be55-039d02372b3d admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:47.645941 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bf576f-fef8-453b-be55-039d02372b3d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:47.646264 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bf576f-fef8-453b-be55-039d02372b3d admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:47.646964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3bf576f-fef8-453b-be55-039d02372b3d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:47.649290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3bf576f-fef8-453b-be55-039d02372b3d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:47.694354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 809/1636] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:47 2022] POST /identity/v3/users => generated 287 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:57:47.695698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e35b660-f8e1-4e9e-8a41-057501a00f2d admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:47.706789 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e35b660-f8e1-4e9e-8a41-057501a00f2d admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:47.713306 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e65e4af-c50b-45b2-bf40-25e0b9cb83b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:47.714032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e65e4af-c50b-45b2-bf40-25e0b9cb83b0 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:47.714411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e65e4af-c50b-45b2-bf40-25e0b9cb83b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:47.714756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e65e4af-c50b-45b2-bf40-25e0b9cb83b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:47.715588 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e65e4af-c50b-45b2-bf40-25e0b9cb83b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:47.717401 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e65e4af-c50b-45b2-bf40-25e0b9cb83b0 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:47.729402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 828/1637] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:47 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:47.733468 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3bf576f-fef8-453b-be55-039d02372b3d admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:47.746225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3bf576f-fef8-453b-be55-039d02372b3d admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:47.751803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5531f1ad-ef4f-4b37-9c45-c634cc4b0cd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:47.752227 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5531f1ad-ef4f-4b37-9c45-c634cc4b0cd6 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:47.752509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5531f1ad-ef4f-4b37-9c45-c634cc4b0cd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:47.752784 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5531f1ad-ef4f-4b37-9c45-c634cc4b0cd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:47.753437 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5531f1ad-ef4f-4b37-9c45-c634cc4b0cd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:47.755502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5531f1ad-ef4f-4b37-9c45-c634cc4b0cd6 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:47.769393 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 810/1638] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:47 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:47.774049 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e65e4af-c50b-45b2-bf40-25e0b9cb83b0 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:47.785004 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e65e4af-c50b-45b2-bf40-25e0b9cb83b0 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:47.790290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec04e70b-7a8d-417b-a5f7-d089bf811e35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:47.791529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec04e70b-7a8d-417b-a5f7-d089bf811e35 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:47.791899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec04e70b-7a8d-417b-a5f7-d089bf811e35 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:47.792160 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec04e70b-7a8d-417b-a5f7-d089bf811e35 admin admin] PATH_INFO: `/v3/projects/b862ff186521426dbf0da8a38198d645/users/2b003dae54c74a9784b1d76e8388a2c0/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:47.810275 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec04e70b-7a8d-417b-a5f7-d089bf811e35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b862ff186521426dbf0da8a38198d645, user_id=2b003dae54c74a9784b1d76e8388a2c0, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:47.813590 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec04e70b-7a8d-417b-a5f7-d089bf811e35 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:47.831060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 829/1639] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:47 2022] PUT /identity/v3/projects/b862ff186521426dbf0da8a38198d645/users/2b003dae54c74a9784b1d76e8388a2c0/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:47.836428 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c905d07-b2a0-4039-ad8a-e212c7879f4f None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:47.836664 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c905d07-b2a0-4039-ad8a-e212c7879f4f None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:47.837010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c905d07-b2a0-4039-ad8a-e212c7879f4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:47.869155 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-9c905d07-b2a0-4039-ad8a-e212c7879f4f None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:47.870377 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-9c905d07-b2a0-4039-ad8a-e212c7879f4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:47.933247 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 811/1640] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:47.943530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec04e70b-7a8d-417b-a5f7-d089bf811e35 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:47.945834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ec04e70b-7a8d-417b-a5f7-d089bf811e35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:48.022769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec04e70b-7a8d-417b-a5f7-d089bf811e35 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:48.027853 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a047611b-4474-4f11-895b-de586c6e3474 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:48.028638 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a047611b-4474-4f11-895b-de586c6e3474 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:48.029087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a047611b-4474-4f11-895b-de586c6e3474 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:48.029982 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a047611b-4474-4f11-895b-de586c6e3474 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:48.056845 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a047611b-4474-4f11-895b-de586c6e3474 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:48.058290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a047611b-4474-4f11-895b-de586c6e3474 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:48.069483 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 830/1641] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:48.691279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14540d29-d5cf-4bbd-98bb-f775d79d3384 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:48.692047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14540d29-d5cf-4bbd-98bb-f775d79d3384 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:48.692329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14540d29-d5cf-4bbd-98bb-f775d79d3384 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:48.751607 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-14540d29-d5cf-4bbd-98bb-f775d79d3384 None None] MFA Rules not processed for user `2b003dae54c74a9784b1d76e8388a2c0`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:48.757133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-14540d29-d5cf-4bbd-98bb-f775d79d3384 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:48.931431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 812/1642] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:48 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:48.937113 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a047611b-4474-4f11-895b-de586c6e3474 service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:48.947966 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a047611b-4474-4f11-895b-de586c6e3474 service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:48.962256 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f374cd7-1349-4d2c-91d6-bc7a2c50eeda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:48.964168 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f374cd7-1349-4d2c-91d6-bc7a2c50eeda admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:48.964692 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f374cd7-1349-4d2c-91d6-bc7a2c50eeda admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:48.966309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f374cd7-1349-4d2c-91d6-bc7a2c50eeda admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:48.971259 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f374cd7-1349-4d2c-91d6-bc7a2c50eeda admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:48.973575 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f374cd7-1349-4d2c-91d6-bc7a2c50eeda admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:49.051151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 831/1643] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:48 2022] POST /identity/v3/projects => generated 371 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:49.052321 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14540d29-d5cf-4bbd-98bb-f775d79d3384 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:49.069932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14540d29-d5cf-4bbd-98bb-f775d79d3384 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:49.086520 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:49.204552 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb303d7-0be1-4744-96d9-fac9fbe4e031 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:49.205270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb303d7-0be1-4744-96d9-fac9fbe4e031 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:49.205649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb303d7-0be1-4744-96d9-fac9fbe4e031 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:49.205981 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb303d7-0be1-4744-96d9-fac9fbe4e031 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:49.206933 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb303d7-0be1-4744-96d9-fac9fbe4e031 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:49.210924 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb303d7-0be1-4744-96d9-fac9fbe4e031 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:49.273636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 813/1644] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:49 2022] POST /identity/v3/users => generated 292 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:49.275065 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f374cd7-1349-4d2c-91d6-bc7a2c50eeda admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:49.299080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f374cd7-1349-4d2c-91d6-bc7a2c50eeda admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:49.314282 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c440182-1c85-4dd0-9502-69ff09805bfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:49.314282 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c440182-1c85-4dd0-9502-69ff09805bfb admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:49.314282 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c440182-1c85-4dd0-9502-69ff09805bfb admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:49.314282 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c440182-1c85-4dd0-9502-69ff09805bfb admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:49.314574 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c440182-1c85-4dd0-9502-69ff09805bfb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:49.314574 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c440182-1c85-4dd0-9502-69ff09805bfb admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:49.327207 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 832/1645] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:49 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:49.328782 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb303d7-0be1-4744-96d9-fac9fbe4e031 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:49.348779 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb303d7-0be1-4744-96d9-fac9fbe4e031 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:49.355760 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988026b4-4268-4b8a-a135-8d9554fcca1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:49.357374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988026b4-4268-4b8a-a135-8d9554fcca1d admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:49.357895 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988026b4-4268-4b8a-a135-8d9554fcca1d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:49.358355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988026b4-4268-4b8a-a135-8d9554fcca1d admin admin] PATH_INFO: `/v3/projects/53440d7b5288482da82c3c3f76d8e759/users/db1ccc457fe944159ef3db1e12c2c9ae/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:49.380727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988026b4-4268-4b8a-a135-8d9554fcca1d admin admin] RBAC: Authorizing `identity:create_grant(project_id=53440d7b5288482da82c3c3f76d8e759, user_id=db1ccc457fe944159ef3db1e12c2c9ae, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:49.383430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988026b4-4268-4b8a-a135-8d9554fcca1d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:49.401607 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 814/1646] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:49 2022] PUT /identity/v3/projects/53440d7b5288482da82c3c3f76d8e759/users/db1ccc457fe944159ef3db1e12c2c9ae/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:49.406258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c440182-1c85-4dd0-9502-69ff09805bfb admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:49.418968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c440182-1c85-4dd0-9502-69ff09805bfb admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:49.508194 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1790839d-8732-4f0a-b122-2a6545894599 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:49.508754 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1790839d-8732-4f0a-b122-2a6545894599 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:49.509366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1790839d-8732-4f0a-b122-2a6545894599 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:49.509661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1790839d-8732-4f0a-b122-2a6545894599 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:49.510291 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1790839d-8732-4f0a-b122-2a6545894599 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:49.511971 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1790839d-8732-4f0a-b122-2a6545894599 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:49.525760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 833/1647] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:49 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:49.531732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988026b4-4268-4b8a-a135-8d9554fcca1d admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:49.546036 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988026b4-4268-4b8a-a135-8d9554fcca1d admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:49.552885 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7004d4ab-3702-4c69-9076-0f7dd4f1999a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:49.553553 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7004d4ab-3702-4c69-9076-0f7dd4f1999a admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:49.553837 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7004d4ab-3702-4c69-9076-0f7dd4f1999a admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:49.554114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7004d4ab-3702-4c69-9076-0f7dd4f1999a admin admin] PATH_INFO: `/v3/projects/53440d7b5288482da82c3c3f76d8e759/users/db1ccc457fe944159ef3db1e12c2c9ae/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:49.556424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7004d4ab-3702-4c69-9076-0f7dd4f1999a admin admin] RBAC: Authorizing `identity:create_grant(project_id=53440d7b5288482da82c3c3f76d8e759, user_id=db1ccc457fe944159ef3db1e12c2c9ae, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:49.559089 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7004d4ab-3702-4c69-9076-0f7dd4f1999a admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:49.571092 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 815/1648] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:49 2022] PUT /identity/v3/projects/53440d7b5288482da82c3c3f76d8e759/users/db1ccc457fe944159ef3db1e12c2c9ae/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:50.506697 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a62786-f7e4-44b2-aa2b-9839104230de None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:50.507511 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a62786-f7e4-44b2-aa2b-9839104230de None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:50.508105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a62786-f7e4-44b2-aa2b-9839104230de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:50.617439 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-75a62786-f7e4-44b2-aa2b-9839104230de None None] MFA Rules not processed for user `db1ccc457fe944159ef3db1e12c2c9ae`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:50.636432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-75a62786-f7e4-44b2-aa2b-9839104230de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:50.667387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:50.781591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 834/1649] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:50 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:50.810354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7004d4ab-3702-4c69-9076-0f7dd4f1999a admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:50.812481 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-7004d4ab-3702-4c69-9076-0f7dd4f1999a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:50.848807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a62786-f7e4-44b2-aa2b-9839104230de None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:50.850817 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-75a62786-f7e4-44b2-aa2b-9839104230de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:50.946361 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7004d4ab-3702-4c69-9076-0f7dd4f1999a admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:50.950417 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4cddc7-21d0-4a9f-8b8b-3c445747e6bb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:50.951088 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4cddc7-21d0-4a9f-8b8b-3c445747e6bb service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:50.951373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4cddc7-21d0-4a9f-8b8b-3c445747e6bb service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:50.951677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4cddc7-21d0-4a9f-8b8b-3c445747e6bb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:50.965819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a62786-f7e4-44b2-aa2b-9839104230de None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:50.971497 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b46ef17-f572-44a0-9ed5-c719a99e08f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:50.972389 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b46ef17-f572-44a0-9ed5-c719a99e08f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:50.972502 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b46ef17-f572-44a0-9ed5-c719a99e08f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:50.973141 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b46ef17-f572-44a0-9ed5-c719a99e08f8 admin admin] PATH_INFO: `/v3/users/5f5b9837682e492997c24009d9526e4d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:50.974007 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b46ef17-f572-44a0-9ed5-c719a99e08f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5f5b9837682e492997c24009d9526e4d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:50.976324 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b46ef17-f572-44a0-9ed5-c719a99e08f8 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:51.036782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8b46ef17-f572-44a0-9ed5-c719a99e08f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5f5b9837682e492997c24009d9526e4d'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:51.037445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8b46ef17-f572-44a0-9ed5-c719a99e08f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5f5b9837682e492997c24009d9526e4d'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:51.043364 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4cddc7-21d0-4a9f-8b8b-3c445747e6bb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:51.045506 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4cddc7-21d0-4a9f-8b8b-3c445747e6bb service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:51.054487 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8b46ef17-f572-44a0-9ed5-c719a99e08f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5f5b9837682e492997c24009d9526e4d'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:51.062456 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8b46ef17-f572-44a0-9ed5-c719a99e08f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5f5b9837682e492997c24009d9526e4d'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:51.070937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 816/1650] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:50 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 261 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:51.072745 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8b46ef17-f572-44a0-9ed5-c719a99e08f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5f5b9837682e492997c24009d9526e4d'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:51.083719 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 835/1651] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:50 2022] DELETE /identity/v3/users/5f5b9837682e492997c24009d9526e4d => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:51.090120 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e52424-57a1-4903-96b4-def1668ea490 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:51.091340 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e52424-57a1-4903-96b4-def1668ea490 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:51.091770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e52424-57a1-4903-96b4-def1668ea490 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:51.115103 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-81e52424-57a1-4903-96b4-def1668ea490 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:51.116522 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-81e52424-57a1-4903-96b4-def1668ea490 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:51.130584 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:51.199513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 817/1652] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:51.209949 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b46ef17-f572-44a0-9ed5-c719a99e08f8 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:51.211456 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8b46ef17-f572-44a0-9ed5-c719a99e08f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:51.254376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:51.302992 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b46ef17-f572-44a0-9ed5-c719a99e08f8 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:51.308570 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bd03ff-3f6f-4d29-9c30-73830ca7a812 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:51.311788 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bd03ff-3f6f-4d29-9c30-73830ca7a812 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:51.312675 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bd03ff-3f6f-4d29-9c30-73830ca7a812 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:51.312675 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bd03ff-3f6f-4d29-9c30-73830ca7a812 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:51.326546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bd03ff-3f6f-4d29-9c30-73830ca7a812 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:51.329726 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bd03ff-3f6f-4d29-9c30-73830ca7a812 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:51.343486 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 836/1653] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:51.596883 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e52424-57a1-4903-96b4-def1668ea490 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:51.598517 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-81e52424-57a1-4903-96b4-def1668ea490 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:51.613877 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e52424-57a1-4903-96b4-def1668ea490 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:51.619824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75dae5a2-c8ea-43f8-98a4-299d705749eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:51.620535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dae5a2-c8ea-43f8-98a4-299d705749eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:51.620886 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dae5a2-c8ea-43f8-98a4-299d705749eb admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:51.621230 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dae5a2-c8ea-43f8-98a4-299d705749eb admin admin] PATH_INFO: `/v3/users/d16822ed88ee49338a24f0ad3aca4f7a` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:51.622565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75dae5a2-c8ea-43f8-98a4-299d705749eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=d16822ed88ee49338a24f0ad3aca4f7a)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:51.624506 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75dae5a2-c8ea-43f8-98a4-299d705749eb admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:51.688291 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-75dae5a2-c8ea-43f8-98a4-299d705749eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd16822ed88ee49338a24f0ad3aca4f7a'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:51.698672 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-75dae5a2-c8ea-43f8-98a4-299d705749eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd16822ed88ee49338a24f0ad3aca4f7a'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:51.699693 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-75dae5a2-c8ea-43f8-98a4-299d705749eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd16822ed88ee49338a24f0ad3aca4f7a'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:51.700568 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-75dae5a2-c8ea-43f8-98a4-299d705749eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd16822ed88ee49338a24f0ad3aca4f7a'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:51.723248 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-75dae5a2-c8ea-43f8-98a4-299d705749eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd16822ed88ee49338a24f0ad3aca4f7a'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:51.737909 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 818/1654] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:51 2022] DELETE /identity/v3/users/d16822ed88ee49338a24f0ad3aca4f7a => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:51.742205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af57f6e5-8305-472b-878c-80f5da521c60 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:51.742681 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af57f6e5-8305-472b-878c-80f5da521c60 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:51.742767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af57f6e5-8305-472b-878c-80f5da521c60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:51.765136 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-af57f6e5-8305-472b-878c-80f5da521c60 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:51.766359 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-af57f6e5-8305-472b-878c-80f5da521c60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:51.831748 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 837/1655] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:51.842041 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75dae5a2-c8ea-43f8-98a4-299d705749eb admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:51.843853 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-75dae5a2-c8ea-43f8-98a4-299d705749eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:51.918772 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75dae5a2-c8ea-43f8-98a4-299d705749eb admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:51.923030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18451c0-a42b-4c2f-9d27-64faf6f39ca7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:51.923775 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18451c0-a42b-4c2f-9d27-64faf6f39ca7 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:51.924126 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18451c0-a42b-4c2f-9d27-64faf6f39ca7 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:51.924510 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18451c0-a42b-4c2f-9d27-64faf6f39ca7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:51.937706 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18451c0-a42b-4c2f-9d27-64faf6f39ca7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:51.939207 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18451c0-a42b-4c2f-9d27-64faf6f39ca7 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:51.949974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 819/1656] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:52.035254 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af57f6e5-8305-472b-878c-80f5da521c60 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:52.044855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-af57f6e5-8305-472b-878c-80f5da521c60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:52.058229 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af57f6e5-8305-472b-878c-80f5da521c60 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:52.067379 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:52.068511 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:52.070220 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:52.070705 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7 admin admin] PATH_INFO: `/v3/projects/5575957500cd4f84b4d65520cece20aa` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:52.073354 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5575957500cd4f84b4d65520cece20aa)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:52.080507 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:52.191441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5575957500cd4f84b4d65520cece20aa', 'request_id': 'req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.205524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5575957500cd4f84b4d65520cece20aa', 'request_id': 'req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.219504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5575957500cd4f84b4d65520cece20aa', 'request_id': 'req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.235284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7 admin admin] The token cache is being invalidate because project 5575957500cd4f84b4d65520cece20aa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:52.235626 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.239323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.241218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 838/1657] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:52 2022] DELETE /identity/v3/projects/5575957500cd4f84b4d65520cece20aa => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:52.246345 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18451c0-a42b-4c2f-9d27-64faf6f39ca7 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:52.257035 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18451c0-a42b-4c2f-9d27-64faf6f39ca7 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:52.303814 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:52.334760 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f3f99c-df08-489c-ad1d-6dd7f971fcd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:52.335662 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f3f99c-df08-489c-ad1d-6dd7f971fcd7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:52.335949 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f3f99c-df08-489c-ad1d-6dd7f971fcd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:52.336221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f3f99c-df08-489c-ad1d-6dd7f971fcd7 admin admin] PATH_INFO: `/v3/users/7f61ce7317814d2b941ca3f064cfda6f` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:52.337840 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f3f99c-df08-489c-ad1d-6dd7f971fcd7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f61ce7317814d2b941ca3f064cfda6f)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:52.339826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f3f99c-df08-489c-ad1d-6dd7f971fcd7 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:52.396648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-52f3f99c-df08-489c-ad1d-6dd7f971fcd7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f61ce7317814d2b941ca3f064cfda6f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.407961 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-52f3f99c-df08-489c-ad1d-6dd7f971fcd7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f61ce7317814d2b941ca3f064cfda6f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.417701 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-52f3f99c-df08-489c-ad1d-6dd7f971fcd7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f61ce7317814d2b941ca3f064cfda6f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.418618 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-52f3f99c-df08-489c-ad1d-6dd7f971fcd7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f61ce7317814d2b941ca3f064cfda6f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.435370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-52f3f99c-df08-489c-ad1d-6dd7f971fcd7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f61ce7317814d2b941ca3f064cfda6f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.443241 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:52.444722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:52.452440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 820/1658] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:52 2022] DELETE /identity/v3/users/7f61ce7317814d2b941ca3f064cfda6f => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:52.478159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66679a3-ff03-4815-bffe-1b4b6ce5d8a7 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:52.578545 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:52.579326 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:52.579636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:52.579920 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d admin admin] PATH_INFO: `/v3/projects/cf9ce4b4c913434ab045fd477358afc2` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:52.581243 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d admin admin] RBAC: Authorizing `identity:delete_project(project_id=cf9ce4b4c913434ab045fd477358afc2)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:52.584644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:52.586320 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:52.627585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f3f99c-df08-489c-ad1d-6dd7f971fcd7 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:52.629736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-52f3f99c-df08-489c-ad1d-6dd7f971fcd7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:52.647870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cf9ce4b4c913434ab045fd477358afc2', 'request_id': 'req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.656263 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cf9ce4b4c913434ab045fd477358afc2', 'request_id': 'req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.664481 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cf9ce4b4c913434ab045fd477358afc2', 'request_id': 'req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.688686 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d admin admin] The token cache is being invalidate because project cf9ce4b4c913434ab045fd477358afc2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:52.689262 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.690571 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.692842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 839/1659] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:52 2022] DELETE /identity/v3/projects/cf9ce4b4c913434ab045fd477358afc2 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:52.699632 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:52.704041 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:52.720114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f3f99c-df08-489c-ad1d-6dd7f971fcd7 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:52.780217 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308be915-6228-45e0-bc8a-b2cfaecad62a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:52.780881 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308be915-6228-45e0-bc8a-b2cfaecad62a service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:52.781174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308be915-6228-45e0-bc8a-b2cfaecad62a service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:52.781735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308be915-6228-45e0-bc8a-b2cfaecad62a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:52.783713 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-308be915-6228-45e0-bc8a-b2cfaecad62a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:52.793023 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0129d8-f92c-42af-8fec-d9ae0eb2723d admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:52.798540 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ab0407-ae07-44c0-914d-8de4ba8fcdd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:52.799192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ab0407-ae07-44c0-914d-8de4ba8fcdd5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:52.799471 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ab0407-ae07-44c0-914d-8de4ba8fcdd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:52.799784 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ab0407-ae07-44c0-914d-8de4ba8fcdd5 admin admin] PATH_INFO: `/v3/users/be65a1124e934b25b277144dc05c2099` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:52.801934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ab0407-ae07-44c0-914d-8de4ba8fcdd5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=be65a1124e934b25b277144dc05c2099)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:52.804185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ab0407-ae07-44c0-914d-8de4ba8fcdd5 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:52.815649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:52.870019 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-53ab0407-ae07-44c0-914d-8de4ba8fcdd5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'be65a1124e934b25b277144dc05c2099'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.870878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-53ab0407-ae07-44c0-914d-8de4ba8fcdd5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'be65a1124e934b25b277144dc05c2099'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.882765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308be915-6228-45e0-bc8a-b2cfaecad62a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:52.884275 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308be915-6228-45e0-bc8a-b2cfaecad62a service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:52.891853 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-53ab0407-ae07-44c0-914d-8de4ba8fcdd5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'be65a1124e934b25b277144dc05c2099'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.896907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 821/1660] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:52 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 270 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:57:52.900542 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308be915-6228-45e0-bc8a-b2cfaecad62a service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:52.903536 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-53ab0407-ae07-44c0-914d-8de4ba8fcdd5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'be65a1124e934b25b277144dc05c2099'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.906716 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-308be915-6228-45e0-bc8a-b2cfaecad62a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:52.940082 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-53ab0407-ae07-44c0-914d-8de4ba8fcdd5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'be65a1124e934b25b277144dc05c2099'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:52.952052 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 840/1661] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:52 2022] DELETE /identity/v3/users/be65a1124e934b25b277144dc05c2099 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:52.956791 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308be915-6228-45e0-bc8a-b2cfaecad62a service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:53.152648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f836d32-7c2f-4471-ae94-15fd2a4404b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:53.152648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f836d32-7c2f-4471-ae94-15fd2a4404b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:53.152648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f836d32-7c2f-4471-ae94-15fd2a4404b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:53.152648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f836d32-7c2f-4471-ae94-15fd2a4404b3 admin admin] PATH_INFO: `/v3/projects/32dd93585caa492f9d38a993db66a468` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:53.153601 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f836d32-7c2f-4471-ae94-15fd2a4404b3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=32dd93585caa492f9d38a993db66a468)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:53.161112 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f836d32-7c2f-4471-ae94-15fd2a4404b3 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:53.324734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0f836d32-7c2f-4471-ae94-15fd2a4404b3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '32dd93585caa492f9d38a993db66a468', 'request_id': 'req-0f836d32-7c2f-4471-ae94-15fd2a4404b3'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:53.365439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0f836d32-7c2f-4471-ae94-15fd2a4404b3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '32dd93585caa492f9d38a993db66a468', 'request_id': 'req-0f836d32-7c2f-4471-ae94-15fd2a4404b3'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:53.376798 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0f836d32-7c2f-4471-ae94-15fd2a4404b3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '32dd93585caa492f9d38a993db66a468', 'request_id': 'req-0f836d32-7c2f-4471-ae94-15fd2a4404b3'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:53.404575 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0f836d32-7c2f-4471-ae94-15fd2a4404b3 admin admin] The token cache is being invalidate because project 32dd93585caa492f9d38a993db66a468 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:53.404881 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0f836d32-7c2f-4471-ae94-15fd2a4404b3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:53.405853 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0f836d32-7c2f-4471-ae94-15fd2a4404b3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:53.408540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 822/1662] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:52 2022] DELETE /identity/v3/projects/32dd93585caa492f9d38a993db66a468 => generated 0 bytes in 509 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:53.413698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ab0407-ae07-44c0-914d-8de4ba8fcdd5 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:53.416468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-53ab0407-ae07-44c0-914d-8de4ba8fcdd5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:53.435952 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ab0407-ae07-44c0-914d-8de4ba8fcdd5 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:53.444586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:53.609264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7670523b-f788-4d82-8628-4850b5d677e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:53.609908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7670523b-f788-4d82-8628-4850b5d677e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:53.610195 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7670523b-f788-4d82-8628-4850b5d677e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:53.610471 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7670523b-f788-4d82-8628-4850b5d677e3 admin admin] PATH_INFO: `/v3/users/03c825567068473eb8bbdcdcf3f70861` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:53.621574 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7670523b-f788-4d82-8628-4850b5d677e3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=03c825567068473eb8bbdcdcf3f70861)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:53.624851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7670523b-f788-4d82-8628-4850b5d677e3 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:53.696474 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7670523b-f788-4d82-8628-4850b5d677e3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '03c825567068473eb8bbdcdcf3f70861'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:53.697333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7670523b-f788-4d82-8628-4850b5d677e3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '03c825567068473eb8bbdcdcf3f70861'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:53.719817 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7670523b-f788-4d82-8628-4850b5d677e3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '03c825567068473eb8bbdcdcf3f70861'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:53.733028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7670523b-f788-4d82-8628-4850b5d677e3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '03c825567068473eb8bbdcdcf3f70861'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:53.747060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7670523b-f788-4d82-8628-4850b5d677e3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '03c825567068473eb8bbdcdcf3f70861'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:53.764208 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 841/1663] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:53 2022] DELETE /identity/v3/users/03c825567068473eb8bbdcdcf3f70861 => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:53.798518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f836d32-7c2f-4471-ae94-15fd2a4404b3 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:53.800197 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-0f836d32-7c2f-4471-ae94-15fd2a4404b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:53.816531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f836d32-7c2f-4471-ae94-15fd2a4404b3 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:53.892912 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:53.911757 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbad0d5-946b-43a7-91bb-7e8fd50801a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:53.912805 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbad0d5-946b-43a7-91bb-7e8fd50801a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:53.913285 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbad0d5-946b-43a7-91bb-7e8fd50801a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:53.913764 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbad0d5-946b-43a7-91bb-7e8fd50801a5 admin admin] PATH_INFO: `/v3/projects/b6b3250de59043739c02fb6839b0be73` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:53.916402 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbad0d5-946b-43a7-91bb-7e8fd50801a5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b6b3250de59043739c02fb6839b0be73)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:53.919098 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbad0d5-946b-43a7-91bb-7e8fd50801a5 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:53.976175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-abbad0d5-946b-43a7-91bb-7e8fd50801a5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b6b3250de59043739c02fb6839b0be73', 'request_id': 'req-abbad0d5-946b-43a7-91bb-7e8fd50801a5'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:53.985566 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-abbad0d5-946b-43a7-91bb-7e8fd50801a5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b6b3250de59043739c02fb6839b0be73', 'request_id': 'req-abbad0d5-946b-43a7-91bb-7e8fd50801a5'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:53.993104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-abbad0d5-946b-43a7-91bb-7e8fd50801a5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b6b3250de59043739c02fb6839b0be73', 'request_id': 'req-abbad0d5-946b-43a7-91bb-7e8fd50801a5'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:54.005102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-abbad0d5-946b-43a7-91bb-7e8fd50801a5 admin admin] The token cache is being invalidate because project b6b3250de59043739c02fb6839b0be73 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:54.005398 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-abbad0d5-946b-43a7-91bb-7e8fd50801a5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:54.006114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-abbad0d5-946b-43a7-91bb-7e8fd50801a5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:54.011671 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 823/1664] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:53 2022] DELETE /identity/v3/projects/b6b3250de59043739c02fb6839b0be73 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:54.013218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27de8c1b-771d-4029-a707-84b08a82b9da None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:54.013721 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27de8c1b-771d-4029-a707-84b08a82b9da None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:54.013801 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27de8c1b-771d-4029-a707-84b08a82b9da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:54.034965 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-27de8c1b-771d-4029-a707-84b08a82b9da None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:54.036590 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-27de8c1b-771d-4029-a707-84b08a82b9da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:54.107781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 842/1665] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:57:54.116806 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c282ab31-2027-4d19-81a0-1326571d10c6 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:54.117133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c282ab31-2027-4d19-81a0-1326571d10c6 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:54.117461 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c282ab31-2027-4d19-81a0-1326571d10c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:54.143033 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-c282ab31-2027-4d19-81a0-1326571d10c6 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:54.144461 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-c282ab31-2027-4d19-81a0-1326571d10c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:54.160022 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 824/1666] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:54.164068 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27de8c1b-771d-4029-a707-84b08a82b9da None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:54.176540 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27de8c1b-771d-4029-a707-84b08a82b9da None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:54.183179 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c751f28c-1486-440a-850f-caf0d31d11aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:54.184188 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c751f28c-1486-440a-850f-caf0d31d11aa admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:54.184549 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c751f28c-1486-440a-850f-caf0d31d11aa admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:54.185006 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c751f28c-1486-440a-850f-caf0d31d11aa admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:54.185943 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c751f28c-1486-440a-850f-caf0d31d11aa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:57:54.186326 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c751f28c-1486-440a-850f-caf0d31d11aa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:54.188302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c751f28c-1486-440a-850f-caf0d31d11aa admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:54.209761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 843/1667] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:57:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:57:54.210877 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c282ab31-2027-4d19-81a0-1326571d10c6 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:54.230977 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c282ab31-2027-4d19-81a0-1326571d10c6 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:54.239160 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d339c2f-7611-47a6-99b9-94acb0a39dd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:54.239982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d339c2f-7611-47a6-99b9-94acb0a39dd8 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:54.240267 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d339c2f-7611-47a6-99b9-94acb0a39dd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:54.240541 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d339c2f-7611-47a6-99b9-94acb0a39dd8 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:54.241354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d339c2f-7611-47a6-99b9-94acb0a39dd8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:54.243779 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d339c2f-7611-47a6-99b9-94acb0a39dd8 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:54.279095 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 825/1668] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:54 2022] POST /identity/v3/projects => generated 381 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:54.284462 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c751f28c-1486-440a-850f-caf0d31d11aa admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:54.297590 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c751f28c-1486-440a-850f-caf0d31d11aa admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:54.319167 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:54.330164 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d339c2f-7611-47a6-99b9-94acb0a39dd8 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:54.332693 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-8d339c2f-7611-47a6-99b9-94acb0a39dd8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:54.428561 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512d13ee-7c50-4d70-851a-d234686f0b41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:54.429413 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512d13ee-7c50-4d70-851a-d234686f0b41 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:54.429773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512d13ee-7c50-4d70-851a-d234686f0b41 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:54.430126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512d13ee-7c50-4d70-851a-d234686f0b41 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:54.431270 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512d13ee-7c50-4d70-851a-d234686f0b41 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:54.433845 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512d13ee-7c50-4d70-851a-d234686f0b41 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:54.476802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 844/1669] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:54 2022] POST /identity/v3/users => generated 291 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:54.478462 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512d13ee-7c50-4d70-851a-d234686f0b41 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:54.479392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d339c2f-7611-47a6-99b9-94acb0a39dd8 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:54.484898 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65990b7a-524b-457b-a873-f6e37b10a28e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:54.485818 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65990b7a-524b-457b-a873-f6e37b10a28e admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:54.487005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65990b7a-524b-457b-a873-f6e37b10a28e admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:54.487334 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65990b7a-524b-457b-a873-f6e37b10a28e admin admin] PATH_INFO: `/v3/projects/aaf529402e634ed89dc6d4fbf8b0696a` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:54.488616 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65990b7a-524b-457b-a873-f6e37b10a28e admin admin] RBAC: Authorizing `identity:delete_project(project_id=aaf529402e634ed89dc6d4fbf8b0696a)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:54.489766 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512d13ee-7c50-4d70-851a-d234686f0b41 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:54.490742 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65990b7a-524b-457b-a873-f6e37b10a28e admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:54.495717 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef6a483-0c36-45c1-96b3-15da4b9f5a00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:54.496344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef6a483-0c36-45c1-96b3-15da4b9f5a00 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:54.496747 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef6a483-0c36-45c1-96b3-15da4b9f5a00 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:54.497034 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef6a483-0c36-45c1-96b3-15da4b9f5a00 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:54.497691 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef6a483-0c36-45c1-96b3-15da4b9f5a00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:54.499299 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef6a483-0c36-45c1-96b3-15da4b9f5a00 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:54.513028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 845/1670] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:54 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:54.517348 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef6a483-0c36-45c1-96b3-15da4b9f5a00 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:54.531884 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef6a483-0c36-45c1-96b3-15da4b9f5a00 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:54.539741 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c52cce-5820-49f4-9359-d23c7d4386d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:54.541306 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c52cce-5820-49f4-9359-d23c7d4386d3 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:54.542153 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c52cce-5820-49f4-9359-d23c7d4386d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:54.542451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c52cce-5820-49f4-9359-d23c7d4386d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:54.543172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c52cce-5820-49f4-9359-d23c7d4386d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:54.545017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c52cce-5820-49f4-9359-d23c7d4386d3 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:54.568645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 846/1671] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:54 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:54.569441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c52cce-5820-49f4-9359-d23c7d4386d3 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:54.584091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c52cce-5820-49f4-9359-d23c7d4386d3 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:54.615100 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-65990b7a-524b-457b-a873-f6e37b10a28e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aaf529402e634ed89dc6d4fbf8b0696a', 'request_id': 'req-65990b7a-524b-457b-a873-f6e37b10a28e'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:54.618971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:54.635925 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-65990b7a-524b-457b-a873-f6e37b10a28e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aaf529402e634ed89dc6d4fbf8b0696a', 'request_id': 'req-65990b7a-524b-457b-a873-f6e37b10a28e'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:54.642690 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-65990b7a-524b-457b-a873-f6e37b10a28e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aaf529402e634ed89dc6d4fbf8b0696a', 'request_id': 'req-65990b7a-524b-457b-a873-f6e37b10a28e'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:54.668858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-65990b7a-524b-457b-a873-f6e37b10a28e admin admin] The token cache is being invalidate because project aaf529402e634ed89dc6d4fbf8b0696a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:54.669057 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-65990b7a-524b-457b-a873-f6e37b10a28e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:54.672272 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-65990b7a-524b-457b-a873-f6e37b10a28e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:54.678561 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 826/1672] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:54 2022] DELETE /identity/v3/projects/aaf529402e634ed89dc6d4fbf8b0696a => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:54.679646 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb29acd-ada1-4c9e-9e70-9c3868140893 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:54.680290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a52d280-9d8f-485e-a045-85e952d4ec17 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:54.681203 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a52d280-9d8f-485e-a045-85e952d4ec17 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:54.681486 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb29acd-ada1-4c9e-9e70-9c3868140893 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:54.681594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a52d280-9d8f-485e-a045-85e952d4ec17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:54.681772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb29acd-ada1-4c9e-9e70-9c3868140893 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:54.682176 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb29acd-ada1-4c9e-9e70-9c3868140893 admin admin] PATH_INFO: `/v3/projects/493156d70b964b3f84a0b673d7e59599/users/d2953471df4a4edd8850c8caf7aa286f/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:54.704942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb29acd-ada1-4c9e-9e70-9c3868140893 admin admin] RBAC: Authorizing `identity:create_grant(project_id=493156d70b964b3f84a0b673d7e59599, user_id=d2953471df4a4edd8850c8caf7aa286f, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:54.708587 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb29acd-ada1-4c9e-9e70-9c3868140893 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:54.716953 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-4a52d280-9d8f-485e-a045-85e952d4ec17 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:54.718292 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-4a52d280-9d8f-485e-a045-85e952d4ec17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:54.734094 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 827/1673] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:57:54.735293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 847/1674] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:54 2022] PUT /identity/v3/projects/493156d70b964b3f84a0b673d7e59599/users/d2953471df4a4edd8850c8caf7aa286f/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:54.745772 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae6795b-d2a5-4dfe-86f9-1bbacff9d5ab None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:54.745772 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae6795b-d2a5-4dfe-86f9-1bbacff9d5ab None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:54.745772 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae6795b-d2a5-4dfe-86f9-1bbacff9d5ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:54.749303 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfec4f80-d9dd-4db4-95d2-f6e016f2f21f None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:54.749821 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfec4f80-d9dd-4db4-95d2-f6e016f2f21f None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:54.750319 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfec4f80-d9dd-4db4-95d2-f6e016f2f21f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:54.791717 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-bfec4f80-d9dd-4db4-95d2-f6e016f2f21f None None] MFA Rules not processed for user `d2953471df4a4edd8850c8caf7aa286f`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:54.793627 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-bfec4f80-d9dd-4db4-95d2-f6e016f2f21f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:54.805278 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-bae6795b-d2a5-4dfe-86f9-1bbacff9d5ab None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:54.807115 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-bae6795b-d2a5-4dfe-86f9-1bbacff9d5ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:54.895116 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:54.908960 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 828/1675] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:54.913336 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae6795b-d2a5-4dfe-86f9-1bbacff9d5ab None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:54.915332 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 848/1676] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:54 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:54.920548 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfec4f80-d9dd-4db4-95d2-f6e016f2f21f None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:54.924158 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-bfec4f80-d9dd-4db4-95d2-f6e016f2f21f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:54.929058 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae6795b-d2a5-4dfe-86f9-1bbacff9d5ab None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:54.951702 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfec4f80-d9dd-4db4-95d2-f6e016f2f21f None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:54.953979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212e2f00-8d66-4f84-8954-1ef5ad0e414a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:54.954870 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212e2f00-8d66-4f84-8954-1ef5ad0e414a admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:54.955323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212e2f00-8d66-4f84-8954-1ef5ad0e414a admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:54.955771 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212e2f00-8d66-4f84-8954-1ef5ad0e414a admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:54.956591 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212e2f00-8d66-4f84-8954-1ef5ad0e414a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:57:54.956899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212e2f00-8d66-4f84-8954-1ef5ad0e414a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:54.958613 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35974e74-4591-4990-84fe-ed31c0f373cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:54.958786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212e2f00-8d66-4f84-8954-1ef5ad0e414a admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:54.959443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35974e74-4591-4990-84fe-ed31c0f373cf admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:54.959822 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35974e74-4591-4990-84fe-ed31c0f373cf admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:54.960163 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35974e74-4591-4990-84fe-ed31c0f373cf admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:54.960874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35974e74-4591-4990-84fe-ed31c0f373cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:54.963833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35974e74-4591-4990-84fe-ed31c0f373cf admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:54.973425 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 829/1677] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:57:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:57:54.979157 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212e2f00-8d66-4f84-8954-1ef5ad0e414a admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:54.990950 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212e2f00-8d66-4f84-8954-1ef5ad0e414a admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:54.997348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946e5677-e4f6-4be3-a824-74ae302fe0a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:54.997977 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946e5677-e4f6-4be3-a824-74ae302fe0a0 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:54.998263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946e5677-e4f6-4be3-a824-74ae302fe0a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:54.998609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946e5677-e4f6-4be3-a824-74ae302fe0a0 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:54.999185 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946e5677-e4f6-4be3-a824-74ae302fe0a0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:55.000402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 849/1678] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:54 2022] POST /identity/v3/projects => generated 379 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:55.004801 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946e5677-e4f6-4be3-a824-74ae302fe0a0 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:55.005137 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35974e74-4591-4990-84fe-ed31c0f373cf admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:55.041504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 830/1679] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:54 2022] POST /identity/v3/projects => generated 385 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:55.046222 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946e5677-e4f6-4be3-a824-74ae302fe0a0 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:55.083642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35974e74-4591-4990-84fe-ed31c0f373cf admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:55.120046 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946e5677-e4f6-4be3-a824-74ae302fe0a0 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:55.134404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f012393-24ec-49c0-a960-1d25ce86ab57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:55.135032 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f012393-24ec-49c0-a960-1d25ce86ab57 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.135303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f012393-24ec-49c0-a960-1d25ce86ab57 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.135601 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f012393-24ec-49c0-a960-1d25ce86ab57 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.136254 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f012393-24ec-49c0-a960-1d25ce86ab57 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:55.138595 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f012393-24ec-49c0-a960-1d25ce86ab57 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:55.155363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e98cade-6fde-4be0-b842-98ef0b61b5cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:55.156465 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e98cade-6fde-4be0-b842-98ef0b61b5cc admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.157040 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e98cade-6fde-4be0-b842-98ef0b61b5cc admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.157643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e98cade-6fde-4be0-b842-98ef0b61b5cc admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.158301 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e98cade-6fde-4be0-b842-98ef0b61b5cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:55.160313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e98cade-6fde-4be0-b842-98ef0b61b5cc admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:55.162773 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 831/1680] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:55 2022] POST /identity/v3/users => generated 293 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:57:55.167493 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f012393-24ec-49c0-a960-1d25ce86ab57 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:55.179458 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f012393-24ec-49c0-a960-1d25ce86ab57 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:55.185777 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cca8db6-3f42-467b-bf91-76d489359922 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:55.186388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cca8db6-3f42-467b-bf91-76d489359922 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.186806 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cca8db6-3f42-467b-bf91-76d489359922 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.187089 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cca8db6-3f42-467b-bf91-76d489359922 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.190530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 850/1681] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:55 2022] POST /identity/v3/users => generated 296 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:57:55.190674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cca8db6-3f42-467b-bf91-76d489359922 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:55.190674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cca8db6-3f42-467b-bf91-76d489359922 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:55.191598 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e98cade-6fde-4be0-b842-98ef0b61b5cc admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:55.205312 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 832/1682] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:55 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 14:57:55.205520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e98cade-6fde-4be0-b842-98ef0b61b5cc admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:55.206821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cca8db6-3f42-467b-bf91-76d489359922 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:55.209596 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d069d6a0-298b-45c7-9593-92359f05cb15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:55.210373 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d069d6a0-298b-45c7-9593-92359f05cb15 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.210873 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d069d6a0-298b-45c7-9593-92359f05cb15 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.211277 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d069d6a0-298b-45c7-9593-92359f05cb15 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.212702 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d069d6a0-298b-45c7-9593-92359f05cb15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:55.214403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d069d6a0-298b-45c7-9593-92359f05cb15 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:55.220351 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cca8db6-3f42-467b-bf91-76d489359922 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:55.226151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 851/1683] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:55 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 14:57:55.230577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d069d6a0-298b-45c7-9593-92359f05cb15 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:55.232416 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb1050d-5159-450b-aae8-e028eb4a64ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:55.233153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb1050d-5159-450b-aae8-e028eb4a64ff admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.233502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb1050d-5159-450b-aae8-e028eb4a64ff admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.233866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb1050d-5159-450b-aae8-e028eb4a64ff admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.234712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb1050d-5159-450b-aae8-e028eb4a64ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:55.237192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb1050d-5159-450b-aae8-e028eb4a64ff admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:55.240478 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d069d6a0-298b-45c7-9593-92359f05cb15 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:55.245951 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4626f3f2-542f-4cf9-9eac-ba18e8d9f627 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:55.246665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4626f3f2-542f-4cf9-9eac-ba18e8d9f627 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.247031 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4626f3f2-542f-4cf9-9eac-ba18e8d9f627 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.247383 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4626f3f2-542f-4cf9-9eac-ba18e8d9f627 admin admin] PATH_INFO: `/v3/projects/dc94a7f2d2db4bc9804cbb4c1c4f7d1c/users/9854e4059a044846908c2740884f583c/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.254128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 833/1684] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:55 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:55.256303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb1050d-5159-450b-aae8-e028eb4a64ff admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:55.268420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4626f3f2-542f-4cf9-9eac-ba18e8d9f627 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc94a7f2d2db4bc9804cbb4c1c4f7d1c, user_id=9854e4059a044846908c2740884f583c, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:55.270881 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:INSERT=5,keystone:DELETE=3,keystone:SELECT=92 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:55.271131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4626f3f2-542f-4cf9-9eac-ba18e8d9f627 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:55.276863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb1050d-5159-450b-aae8-e028eb4a64ff admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:55.284775 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d0cdcf-ae62-45f0-9845-117b287b949e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:55.285841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d0cdcf-ae62-45f0-9845-117b287b949e admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.286557 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d0cdcf-ae62-45f0-9845-117b287b949e admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.286846 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d0cdcf-ae62-45f0-9845-117b287b949e admin admin] PATH_INFO: `/v3/projects/6ba24f5a0f4b40c498bb15e73bd92515/users/55e1465595804fc4a0b45c0c0ed53a2f/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.291128 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 852/1685] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:55 2022] PUT /identity/v3/projects/dc94a7f2d2db4bc9804cbb4c1c4f7d1c/users/9854e4059a044846908c2740884f583c/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:55.295185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4626f3f2-542f-4cf9-9eac-ba18e8d9f627 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:55.306146 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d0cdcf-ae62-45f0-9845-117b287b949e admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ba24f5a0f4b40c498bb15e73bd92515, user_id=55e1465595804fc4a0b45c0c0ed53a2f, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:55.309397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d0cdcf-ae62-45f0-9845-117b287b949e admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:55.326171 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 834/1686] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:55 2022] PUT /identity/v3/projects/6ba24f5a0f4b40c498bb15e73bd92515/users/55e1465595804fc4a0b45c0c0ed53a2f/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:55.338027 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd09bdf-e282-4243-998a-8c4436ece40d None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.338325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd09bdf-e282-4243-998a-8c4436ece40d None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.338597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd09bdf-e282-4243-998a-8c4436ece40d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.340764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:55.370226 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4626f3f2-542f-4cf9-9eac-ba18e8d9f627 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:55.372361 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-5bd09bdf-e282-4243-998a-8c4436ece40d None None] MFA Rules not processed for user `55e1465595804fc4a0b45c0c0ed53a2f`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:55.374051 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5bd09bdf-e282-4243-998a-8c4436ece40d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:55.443934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604de4d9-e6eb-4287-8e78-4f900527055e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:55.445094 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604de4d9-e6eb-4287-8e78-4f900527055e admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.445448 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604de4d9-e6eb-4287-8e78-4f900527055e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.445797 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604de4d9-e6eb-4287-8e78-4f900527055e admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.447398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604de4d9-e6eb-4287-8e78-4f900527055e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:55.449292 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604de4d9-e6eb-4287-8e78-4f900527055e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:55.461422 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 853/1687] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:55 2022] GET /identity/v3/roles => generated 1420 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:55.465388 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604de4d9-e6eb-4287-8e78-4f900527055e admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:55.482342 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 835/1688] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:55 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:57:55.486078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604de4d9-e6eb-4287-8e78-4f900527055e admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:55.488727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd09bdf-e282-4243-998a-8c4436ece40d None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:55.491124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782ec5e9-b539-45ca-a94f-6900ec20fa44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:55.491880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782ec5e9-b539-45ca-a94f-6900ec20fa44 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.492161 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782ec5e9-b539-45ca-a94f-6900ec20fa44 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.492479 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782ec5e9-b539-45ca-a94f-6900ec20fa44 admin admin] PATH_INFO: `/v3/projects/dc94a7f2d2db4bc9804cbb4c1c4f7d1c/users/9854e4059a044846908c2740884f583c/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.495356 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782ec5e9-b539-45ca-a94f-6900ec20fa44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc94a7f2d2db4bc9804cbb4c1c4f7d1c, user_id=9854e4059a044846908c2740884f583c, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:55.498118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782ec5e9-b539-45ca-a94f-6900ec20fa44 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:55.504035 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd09bdf-e282-4243-998a-8c4436ece40d None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:55.511306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6827ce9a-8eed-4918-a5fb-2a27dbabd361 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:55.512282 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6827ce9a-8eed-4918-a5fb-2a27dbabd361 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.512567 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6827ce9a-8eed-4918-a5fb-2a27dbabd361 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.512845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6827ce9a-8eed-4918-a5fb-2a27dbabd361 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.513495 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6827ce9a-8eed-4918-a5fb-2a27dbabd361 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:55.515428 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6827ce9a-8eed-4918-a5fb-2a27dbabd361 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:55.520214 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 854/1689] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:55 2022] PUT /identity/v3/projects/dc94a7f2d2db4bc9804cbb4c1c4f7d1c/users/9854e4059a044846908c2740884f583c/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:55.535320 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d44dcc9-4e97-4f98-88da-006dfc6071a7 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.535723 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d44dcc9-4e97-4f98-88da-006dfc6071a7 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.536236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d44dcc9-4e97-4f98-88da-006dfc6071a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.554345 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 836/1690] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:55 2022] POST /identity/v3/projects => generated 383 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:55.558813 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6827ce9a-8eed-4918-a5fb-2a27dbabd361 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:55.588283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-7d44dcc9-4e97-4f98-88da-006dfc6071a7 None None] MFA Rules not processed for user `9854e4059a044846908c2740884f583c`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:55.589832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7d44dcc9-4e97-4f98-88da-006dfc6071a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:55.641268 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6827ce9a-8eed-4918-a5fb-2a27dbabd361 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:55.647573 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace4b2bf-c411-4550-a7c1-e492f18400bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:55.648279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace4b2bf-c411-4550-a7c1-e492f18400bb admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.648656 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace4b2bf-c411-4550-a7c1-e492f18400bb admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.649004 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace4b2bf-c411-4550-a7c1-e492f18400bb admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.650766 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace4b2bf-c411-4550-a7c1-e492f18400bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:55.652816 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace4b2bf-c411-4550-a7c1-e492f18400bb admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:55.678597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 837/1691] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:55 2022] POST /identity/v3/users => generated 292 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:57:55.683599 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 855/1692] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:55 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:55.685880 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace4b2bf-c411-4550-a7c1-e492f18400bb admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:55.694720 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d44dcc9-4e97-4f98-88da-006dfc6071a7 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:55.697307 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace4b2bf-c411-4550-a7c1-e492f18400bb admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:55.699000 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7d44dcc9-4e97-4f98-88da-006dfc6071a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:55.702110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5fc96c-0866-4ca6-9219-47c16dca6145 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:55.702754 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5fc96c-0866-4ca6-9219-47c16dca6145 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.703265 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5fc96c-0866-4ca6-9219-47c16dca6145 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.703265 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5fc96c-0866-4ca6-9219-47c16dca6145 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.703989 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5fc96c-0866-4ca6-9219-47c16dca6145 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:55.705556 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5fc96c-0866-4ca6-9219-47c16dca6145 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:55.720185 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 838/1693] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:55 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:55.721835 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5fc96c-0866-4ca6-9219-47c16dca6145 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:55.738425 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5fc96c-0866-4ca6-9219-47c16dca6145 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:55.751318 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9ad3bb-7e4e-46b3-a2e1-aecabdb7383a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:55.751955 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9ad3bb-7e4e-46b3-a2e1-aecabdb7383a admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.752230 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9ad3bb-7e4e-46b3-a2e1-aecabdb7383a admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.752496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9ad3bb-7e4e-46b3-a2e1-aecabdb7383a admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.753155 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9ad3bb-7e4e-46b3-a2e1-aecabdb7383a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:55.754755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9ad3bb-7e4e-46b3-a2e1-aecabdb7383a admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:55.767594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 839/1694] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:55 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 14:57:55.771026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9ad3bb-7e4e-46b3-a2e1-aecabdb7383a admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:55.784018 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9ad3bb-7e4e-46b3-a2e1-aecabdb7383a admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:55.788405 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b87ca3-d1d6-4963-8e51-df8d9da90ab5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:55.789057 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b87ca3-d1d6-4963-8e51-df8d9da90ab5 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.789335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b87ca3-d1d6-4963-8e51-df8d9da90ab5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.789608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b87ca3-d1d6-4963-8e51-df8d9da90ab5 admin admin] PATH_INFO: `/v3/projects/9b6e9bd8b6c24bd2bb50f8df8ac2e38b/users/019409f45b174140bd7605f7e9ad088d/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.792569 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d44dcc9-4e97-4f98-88da-006dfc6071a7 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:55.796819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e84870-be13-48bb-852c-059453a9ce38 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:55.797636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e84870-be13-48bb-852c-059453a9ce38 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.798087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e84870-be13-48bb-852c-059453a9ce38 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.798694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e84870-be13-48bb-852c-059453a9ce38 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.803092 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:55.806454 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:55.808254 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b87ca3-d1d6-4963-8e51-df8d9da90ab5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b6e9bd8b6c24bd2bb50f8df8ac2e38b, user_id=019409f45b174140bd7605f7e9ad088d, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:55.811417 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b87ca3-d1d6-4963-8e51-df8d9da90ab5 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:55.812862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e84870-be13-48bb-852c-059453a9ce38 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:55.815270 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e84870-be13-48bb-852c-059453a9ce38 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:55.829599 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 856/1695] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:55 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:55.847794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 840/1696] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:55 2022] PUT /identity/v3/projects/9b6e9bd8b6c24bd2bb50f8df8ac2e38b/users/019409f45b174140bd7605f7e9ad088d/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:55.858282 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9918ef-447b-4114-a272-f15643660094 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:55.858795 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9918ef-447b-4114-a272-f15643660094 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:55.859222 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9918ef-447b-4114-a272-f15643660094 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:55.906735 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-fc9918ef-447b-4114-a272-f15643660094 None None] MFA Rules not processed for user `019409f45b174140bd7605f7e9ad088d`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:55.907997 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-fc9918ef-447b-4114-a272-f15643660094 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:55.993892 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 857/1697] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:55 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:55.999683 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b87ca3-d1d6-4963-8e51-df8d9da90ab5 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:56.118850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b87ca3-d1d6-4963-8e51-df8d9da90ab5 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:56.126610 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddf97ac-789a-4fac-ba2b-16a0f8de139a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:56.127633 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddf97ac-789a-4fac-ba2b-16a0f8de139a admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:56.127994 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddf97ac-789a-4fac-ba2b-16a0f8de139a admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:56.128359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddf97ac-789a-4fac-ba2b-16a0f8de139a admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:56.131446 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddf97ac-789a-4fac-ba2b-16a0f8de139a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:56.134657 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddf97ac-789a-4fac-ba2b-16a0f8de139a admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:56.222463 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 841/1698] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:55 2022] POST /identity/v3/projects => generated 385 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:56.226619 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9918ef-447b-4114-a272-f15643660094 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:56.249057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9918ef-447b-4114-a272-f15643660094 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:56.419635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022a2c57-9010-4f71-8378-2e4f53a3c95a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:56.420774 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022a2c57-9010-4f71-8378-2e4f53a3c95a admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:56.421413 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022a2c57-9010-4f71-8378-2e4f53a3c95a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:56.421917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022a2c57-9010-4f71-8378-2e4f53a3c95a admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:56.423990 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022a2c57-9010-4f71-8378-2e4f53a3c95a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:56.427072 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022a2c57-9010-4f71-8378-2e4f53a3c95a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:56.463892 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 858/1699] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:56 2022] POST /identity/v3/users => generated 299 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:56.467481 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddf97ac-789a-4fac-ba2b-16a0f8de139a admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:56.486488 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddf97ac-789a-4fac-ba2b-16a0f8de139a admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:56.493369 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d9bc6c-09ef-4920-8956-2a020f309382 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:56.494267 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d9bc6c-09ef-4920-8956-2a020f309382 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:56.494544 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d9bc6c-09ef-4920-8956-2a020f309382 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:56.494820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d9bc6c-09ef-4920-8956-2a020f309382 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:56.495481 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d9bc6c-09ef-4920-8956-2a020f309382 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:56.497279 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d9bc6c-09ef-4920-8956-2a020f309382 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:56.517934 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 842/1700] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:56 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:56.533581 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022a2c57-9010-4f71-8378-2e4f53a3c95a admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:56.553799 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022a2c57-9010-4f71-8378-2e4f53a3c95a admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:56.564091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd4c686-3ab9-4618-bc5a-cf3a11736b4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:56.565271 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd4c686-3ab9-4618-bc5a-cf3a11736b4d admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:56.566653 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd4c686-3ab9-4618-bc5a-cf3a11736b4d admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:56.566895 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d9bc6c-09ef-4920-8956-2a020f309382 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:56.567624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd4c686-3ab9-4618-bc5a-cf3a11736b4d admin admin] PATH_INFO: `/v3/projects/8146932fa3e3458f914b0b531dcdaa98/users/2a460b856413424687069a18b5c0e347/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:56.581005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d9bc6c-09ef-4920-8956-2a020f309382 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:56.591600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d277c3-8f8e-4dee-bb18-7f295700879f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:56.592532 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d277c3-8f8e-4dee-bb18-7f295700879f admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:56.593048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d277c3-8f8e-4dee-bb18-7f295700879f admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:56.593198 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d277c3-8f8e-4dee-bb18-7f295700879f admin admin] PATH_INFO: `/v3/users/d2953471df4a4edd8850c8caf7aa286f` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:56.595465 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d277c3-8f8e-4dee-bb18-7f295700879f admin admin] RBAC: Authorizing `identity:delete_user(user_id=d2953471df4a4edd8850c8caf7aa286f)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:56.597866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd4c686-3ab9-4618-bc5a-cf3a11736b4d admin admin] RBAC: Authorizing `identity:create_grant(project_id=8146932fa3e3458f914b0b531dcdaa98, user_id=2a460b856413424687069a18b5c0e347, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:56.601261 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd4c686-3ab9-4618-bc5a-cf3a11736b4d admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:56.610956 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d277c3-8f8e-4dee-bb18-7f295700879f admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:56.636497 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 859/1701] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:56 2022] PUT /identity/v3/projects/8146932fa3e3458f914b0b531dcdaa98/users/2a460b856413424687069a18b5c0e347/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:56.640440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd4c686-3ab9-4618-bc5a-cf3a11736b4d admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:56.702157 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-34d277c3-8f8e-4dee-bb18-7f295700879f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd2953471df4a4edd8850c8caf7aa286f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:56.712933 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-34d277c3-8f8e-4dee-bb18-7f295700879f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd2953471df4a4edd8850c8caf7aa286f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:56.713675 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-34d277c3-8f8e-4dee-bb18-7f295700879f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd2953471df4a4edd8850c8caf7aa286f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:56.714398 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-34d277c3-8f8e-4dee-bb18-7f295700879f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd2953471df4a4edd8850c8caf7aa286f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:56.735871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:56.736864 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-34d277c3-8f8e-4dee-bb18-7f295700879f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd2953471df4a4edd8850c8caf7aa286f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:56.762131 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 843/1702] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:56 2022] DELETE /identity/v3/users/d2953471df4a4edd8850c8caf7aa286f => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:56.766635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd4c686-3ab9-4618-bc5a-cf3a11736b4d admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:56.768693 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ca1b7d-09fb-49b1-889e-725c5316faf5 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:56.768975 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ca1b7d-09fb-49b1-889e-725c5316faf5 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:56.769407 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ca1b7d-09fb-49b1-889e-725c5316faf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:56.821756 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-20ca1b7d-09fb-49b1-889e-725c5316faf5 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:56.823083 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-20ca1b7d-09fb-49b1-889e-725c5316faf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:56.873918 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:56.876645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a75c32-358a-4142-88cc-ce4bb7f51a56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:56.877278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a75c32-358a-4142-88cc-ce4bb7f51a56 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:56.877557 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a75c32-358a-4142-88cc-ce4bb7f51a56 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:56.877826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a75c32-358a-4142-88cc-ce4bb7f51a56 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:56.878667 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a75c32-358a-4142-88cc-ce4bb7f51a56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:56.880309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a75c32-358a-4142-88cc-ce4bb7f51a56 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:56.893664 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 860/1703] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:56 2022] GET /identity/v3/roles => generated 1420 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 14:57:56.898537 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a75c32-358a-4142-88cc-ce4bb7f51a56 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:56.901080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-f4a75c32-358a-4142-88cc-ce4bb7f51a56 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:56.917432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a75c32-358a-4142-88cc-ce4bb7f51a56 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:56.923144 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 844/1704] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:56.923305 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14dd709-0c60-41be-a961-19aec262253b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:56.924186 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14dd709-0c60-41be-a961-19aec262253b admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:56.924542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14dd709-0c60-41be-a961-19aec262253b admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:56.924887 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14dd709-0c60-41be-a961-19aec262253b admin admin] PATH_INFO: `/v3/projects/8146932fa3e3458f914b0b531dcdaa98/users/2a460b856413424687069a18b5c0e347/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:56.927874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14dd709-0c60-41be-a961-19aec262253b admin admin] RBAC: Authorizing `identity:create_grant(project_id=8146932fa3e3458f914b0b531dcdaa98, user_id=2a460b856413424687069a18b5c0e347, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:56.934405 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ca1b7d-09fb-49b1-889e-725c5316faf5 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:56.935985 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-20ca1b7d-09fb-49b1-889e-725c5316faf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:56.936168 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14dd709-0c60-41be-a961-19aec262253b admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:56.951428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 861/1705] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:56 2022] PUT /identity/v3/projects/8146932fa3e3458f914b0b531dcdaa98/users/2a460b856413424687069a18b5c0e347/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:56.963429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80718b70-e5fe-4e88-b8d0-85f33a73cbeb None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:56.964504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80718b70-e5fe-4e88-b8d0-85f33a73cbeb None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:56.965137 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80718b70-e5fe-4e88-b8d0-85f33a73cbeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:57.013850 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-80718b70-e5fe-4e88-b8d0-85f33a73cbeb None None] MFA Rules not processed for user `2a460b856413424687069a18b5c0e347`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:57.015641 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-80718b70-e5fe-4e88-b8d0-85f33a73cbeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:57.037310 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ca1b7d-09fb-49b1-889e-725c5316faf5 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:57.116545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fbe25b-3d90-4436-87ce-479ddb2a25f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:57.117367 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fbe25b-3d90-4436-87ce-479ddb2a25f1 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:57.117767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fbe25b-3d90-4436-87ce-479ddb2a25f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:57.118678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fbe25b-3d90-4436-87ce-479ddb2a25f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:57.137876 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 862/1706] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:56 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:57.150502 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80718b70-e5fe-4e88-b8d0-85f33a73cbeb None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:57.152681 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-80718b70-e5fe-4e88-b8d0-85f33a73cbeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:57.202417 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43fbe25b-3d90-4436-87ce-479ddb2a25f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:57.204004 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43fbe25b-3d90-4436-87ce-479ddb2a25f1 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:57.216409 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 845/1707] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 282 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:57:57.224486 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80718b70-e5fe-4e88-b8d0-85f33a73cbeb None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:57.229631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b67c26-5ef7-4fdf-aadc-649097f5ff42 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:57.230257 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b67c26-5ef7-4fdf-aadc-649097f5ff42 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:57.230532 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b67c26-5ef7-4fdf-aadc-649097f5ff42 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:57.230801 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b67c26-5ef7-4fdf-aadc-649097f5ff42 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:57.232569 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e1b67c26-5ef7-4fdf-aadc-649097f5ff42 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:57.252503 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:57.307897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b67c26-5ef7-4fdf-aadc-649097f5ff42 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:57.310723 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b67c26-5ef7-4fdf-aadc-649097f5ff42 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:57.337441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 863/1708] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:57 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:57:57.759960 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fbe25b-3d90-4436-87ce-479ddb2a25f1 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:57.762424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-43fbe25b-3d90-4436-87ce-479ddb2a25f1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:57.785573 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fbe25b-3d90-4436-87ce-479ddb2a25f1 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:57.798451 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3896d2e4-00c5-4d20-8ad3-ce12c3a61fbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:57.800720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3896d2e4-00c5-4d20-8ad3-ce12c3a61fbb admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:57.801001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3896d2e4-00c5-4d20-8ad3-ce12c3a61fbb admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:57.801333 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3896d2e4-00c5-4d20-8ad3-ce12c3a61fbb admin admin] PATH_INFO: `/v3/projects/493156d70b964b3f84a0b673d7e59599` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:57.802813 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3896d2e4-00c5-4d20-8ad3-ce12c3a61fbb admin admin] RBAC: Authorizing `identity:delete_project(project_id=493156d70b964b3f84a0b673d7e59599)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:57.811134 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3896d2e4-00c5-4d20-8ad3-ce12c3a61fbb admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:57.934671 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:57.956848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-3896d2e4-00c5-4d20-8ad3-ce12c3a61fbb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '493156d70b964b3f84a0b673d7e59599', 'request_id': 'req-3896d2e4-00c5-4d20-8ad3-ce12c3a61fbb'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:57.968256 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-3896d2e4-00c5-4d20-8ad3-ce12c3a61fbb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '493156d70b964b3f84a0b673d7e59599', 'request_id': 'req-3896d2e4-00c5-4d20-8ad3-ce12c3a61fbb'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:57.988310 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-3896d2e4-00c5-4d20-8ad3-ce12c3a61fbb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '493156d70b964b3f84a0b673d7e59599', 'request_id': 'req-3896d2e4-00c5-4d20-8ad3-ce12c3a61fbb'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:58.013957 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-3896d2e4-00c5-4d20-8ad3-ce12c3a61fbb admin admin] The token cache is being invalidate because project 493156d70b964b3f84a0b673d7e59599 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:58.014319 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-3896d2e4-00c5-4d20-8ad3-ce12c3a61fbb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:58.015919 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-3896d2e4-00c5-4d20-8ad3-ce12c3a61fbb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:58.018365 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 846/1709] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:57 2022] DELETE /identity/v3/projects/493156d70b964b3f84a0b673d7e59599 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:58.022345 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b67c26-5ef7-4fdf-aadc-649097f5ff42 service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:58.040839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b67c26-5ef7-4fdf-aadc-649097f5ff42 service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:58.156794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31ee3af-5f86-4e24-b1fc-eea912389a85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:58.157978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31ee3af-5f86-4e24-b1fc-eea912389a85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:58.164009 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31ee3af-5f86-4e24-b1fc-eea912389a85 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:58.164317 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31ee3af-5f86-4e24-b1fc-eea912389a85 admin admin] PATH_INFO: `/v3/users/9854e4059a044846908c2740884f583c` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:58.167294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31ee3af-5f86-4e24-b1fc-eea912389a85 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9854e4059a044846908c2740884f583c)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:58.169904 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31ee3af-5f86-4e24-b1fc-eea912389a85 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:58.234041 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a31ee3af-5f86-4e24-b1fc-eea912389a85 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9854e4059a044846908c2740884f583c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:58.235654 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a31ee3af-5f86-4e24-b1fc-eea912389a85 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9854e4059a044846908c2740884f583c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:58.256563 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a31ee3af-5f86-4e24-b1fc-eea912389a85 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9854e4059a044846908c2740884f583c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:58.267763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a31ee3af-5f86-4e24-b1fc-eea912389a85 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9854e4059a044846908c2740884f583c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:58.278696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a31ee3af-5f86-4e24-b1fc-eea912389a85 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9854e4059a044846908c2740884f583c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:58.288183 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 864/1710] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:58 2022] DELETE /identity/v3/users/9854e4059a044846908c2740884f583c => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:58.738437 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef02584-b01e-491b-a378-a1a2d70d8fe4 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:58.738948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef02584-b01e-491b-a378-a1a2d70d8fe4 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:58.739042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef02584-b01e-491b-a378-a1a2d70d8fe4 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:58.740565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 847/1711] 10.222.0.36 () {54 vars in 921 bytes} [Mon Oct 31 14:57:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:57:58.769158 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc537a96-b2b8-4940-a567-33b7b26d0ee4 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:58.770675 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc537a96-b2b8-4940-a567-33b7b26d0ee4 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:58.770796 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc537a96-b2b8-4940-a567-33b7b26d0ee4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:58.793154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-bc537a96-b2b8-4940-a567-33b7b26d0ee4 None None] MFA Rules not processed for user `70ea2aaf529b46dabb70b71f4dcc3860`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:58.794441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-bc537a96-b2b8-4940-a567-33b7b26d0ee4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:58.845390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:58.869925 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef02584-b01e-491b-a378-a1a2d70d8fe4 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:58.872622 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2ef02584-b01e-491b-a378-a1a2d70d8fe4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:58.915860 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 865/1712] 10.222.0.36 () {58 vars in 1018 bytes} [Mon Oct 31 14:57:58 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:58.929466 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e8d415-f88c-4d64-9af3-740a52703d2e None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:58.929752 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e8d415-f88c-4d64-9af3-740a52703d2e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:58.930020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e8d415-f88c-4d64-9af3-740a52703d2e None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:58.931497 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 866/1713] 10.222.0.36 () {54 vars in 961 bytes} [Mon Oct 31 14:57:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:57:58.937137 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08abac0b-e141-4e53-9b25-7845c8903c90 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:58.937420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08abac0b-e141-4e53-9b25-7845c8903c90 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:58.937688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08abac0b-e141-4e53-9b25-7845c8903c90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:58.961203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-08abac0b-e141-4e53-9b25-7845c8903c90 None None] MFA Rules not processed for user `70ea2aaf529b46dabb70b71f4dcc3860`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:58.963536 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-08abac0b-e141-4e53-9b25-7845c8903c90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:58.973878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 867/1714] 10.222.0.36 () {58 vars in 1058 bytes} [Mon Oct 31 14:57:58 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:57:58.986381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08abac0b-e141-4e53-9b25-7845c8903c90 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:58.991245 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef02584-b01e-491b-a378-a1a2d70d8fe4 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:59.000066 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08abac0b-e141-4e53-9b25-7845c8903c90 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:59.000358 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98c21a0-bb8a-4451-9bef-53e4358d0b87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:59.001663 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98c21a0-bb8a-4451-9bef-53e4358d0b87 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.002506 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98c21a0-bb8a-4451-9bef-53e4358d0b87 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.003309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98c21a0-bb8a-4451-9bef-53e4358d0b87 admin admin] PATH_INFO: `/v3/users/77af489d5daa4d028b737c5f4efc93db` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.006371 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01291854-a389-43ab-a84e-511a1805d7a8 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70ea2aaf529b46dabb70b71f4dcc3860', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:59.006682 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98c21a0-bb8a-4451-9bef-53e4358d0b87 admin admin] RBAC: Authorizing `identity:delete_user(user_id=77af489d5daa4d028b737c5f4efc93db)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:59.007097 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01291854-a389-43ab-a84e-511a1805d7a8 service placement] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.007495 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01291854-a389-43ab-a84e-511a1805d7a8 service placement] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.007825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01291854-a389-43ab-a84e-511a1805d7a8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.009887 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98c21a0-bb8a-4451-9bef-53e4358d0b87 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:59.020641 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01291854-a389-43ab-a84e-511a1805d7a8 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:59.022482 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01291854-a389-43ab-a84e-511a1805d7a8 service placement] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:59.034276 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 868/1715] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:58 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:57:59.035203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01291854-a389-43ab-a84e-511a1805d7a8 service placement] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:59.036550 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-01291854-a389-43ab-a84e-511a1805d7a8 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:59.064783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01291854-a389-43ab-a84e-511a1805d7a8 service placement] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:59.081570 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472f1910-b48b-477d-9a30-75e773a4c1e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:59.082554 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472f1910-b48b-477d-9a30-75e773a4c1e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.082909 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472f1910-b48b-477d-9a30-75e773a4c1e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.083362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472f1910-b48b-477d-9a30-75e773a4c1e2 admin admin] PATH_INFO: `/v3/projects/dc94a7f2d2db4bc9804cbb4c1c4f7d1c` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.084725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472f1910-b48b-477d-9a30-75e773a4c1e2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc94a7f2d2db4bc9804cbb4c1c4f7d1c)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:59.093976 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472f1910-b48b-477d-9a30-75e773a4c1e2 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:59.104309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c98c21a0-bb8a-4451-9bef-53e4358d0b87 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '77af489d5daa4d028b737c5f4efc93db'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:59.113038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c98c21a0-bb8a-4451-9bef-53e4358d0b87 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '77af489d5daa4d028b737c5f4efc93db'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:59.113955 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c98c21a0-bb8a-4451-9bef-53e4358d0b87 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '77af489d5daa4d028b737c5f4efc93db'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:59.114841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c98c21a0-bb8a-4451-9bef-53e4358d0b87 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '77af489d5daa4d028b737c5f4efc93db'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:59.135101 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c98c21a0-bb8a-4451-9bef-53e4358d0b87 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '77af489d5daa4d028b737c5f4efc93db'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:59.153703 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 848/1716] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:57:58 2022] DELETE /identity/v3/users/77af489d5daa4d028b737c5f4efc93db => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:57:59.157436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98c21a0-bb8a-4451-9bef-53e4358d0b87 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:59.158812 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-c98c21a0-bb8a-4451-9bef-53e4358d0b87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:59.160890 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-472f1910-b48b-477d-9a30-75e773a4c1e2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc94a7f2d2db4bc9804cbb4c1c4f7d1c', 'request_id': 'req-472f1910-b48b-477d-9a30-75e773a4c1e2'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:59.168683 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-472f1910-b48b-477d-9a30-75e773a4c1e2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc94a7f2d2db4bc9804cbb4c1c4f7d1c', 'request_id': 'req-472f1910-b48b-477d-9a30-75e773a4c1e2'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:59.178069 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-472f1910-b48b-477d-9a30-75e773a4c1e2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc94a7f2d2db4bc9804cbb4c1c4f7d1c', 'request_id': 'req-472f1910-b48b-477d-9a30-75e773a4c1e2'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:59.196896 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-472f1910-b48b-477d-9a30-75e773a4c1e2 admin admin] The token cache is being invalidate because project dc94a7f2d2db4bc9804cbb4c1c4f7d1c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:59.197240 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-472f1910-b48b-477d-9a30-75e773a4c1e2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:59.197852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-472f1910-b48b-477d-9a30-75e773a4c1e2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:59.199428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 869/1717] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:59 2022] DELETE /identity/v3/projects/dc94a7f2d2db4bc9804cbb4c1c4f7d1c => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:59.204799 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60104b49-c4ee-43f9-adf9-e8332c6edf4f None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.205205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60104b49-c4ee-43f9-adf9-e8332c6edf4f None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.205514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60104b49-c4ee-43f9-adf9-e8332c6edf4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.229678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:59.231625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-60104b49-c4ee-43f9-adf9-e8332c6edf4f None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:59.232990 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-60104b49-c4ee-43f9-adf9-e8332c6edf4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:59.238056 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98c21a0-bb8a-4451-9bef-53e4358d0b87 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:59.247152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d857f4ce-2e89-4b35-aba6-528dda11130e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:59.248486 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d857f4ce-2e89-4b35-aba6-528dda11130e service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.248486 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d857f4ce-2e89-4b35-aba6-528dda11130e service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.248929 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d857f4ce-2e89-4b35-aba6-528dda11130e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.251585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d857f4ce-2e89-4b35-aba6-528dda11130e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:59.335135 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d857f4ce-2e89-4b35-aba6-528dda11130e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:59.336985 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d857f4ce-2e89-4b35-aba6-528dda11130e service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:59.343387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 870/1718] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:59.346269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7349bc15-d2e1-432f-87e8-0d22dbad92bb None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.346563 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7349bc15-d2e1-432f-87e8-0d22dbad92bb None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.346842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7349bc15-d2e1-432f-87e8-0d22dbad92bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.348162 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 849/1719] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:57:59 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:57:59.385919 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-7349bc15-d2e1-432f-87e8-0d22dbad92bb None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:59.387511 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7349bc15-d2e1-432f-87e8-0d22dbad92bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:59.392863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d857f4ce-2e89-4b35-aba6-528dda11130e service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:59.394747 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d857f4ce-2e89-4b35-aba6-528dda11130e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:59.405318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 871/1720] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:59.409263 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7349bc15-d2e1-432f-87e8-0d22dbad92bb None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:59.421054 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7349bc15-d2e1-432f-87e8-0d22dbad92bb None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:59.422271 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d857f4ce-2e89-4b35-aba6-528dda11130e service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:59.431670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b8086f-dcd5-46e1-8be0-f632ac5c3a57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:59.433565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b8086f-dcd5-46e1-8be0-f632ac5c3a57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.434011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b8086f-dcd5-46e1-8be0-f632ac5c3a57 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.434297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b8086f-dcd5-46e1-8be0-f632ac5c3a57 admin admin] PATH_INFO: `/v3/projects/af86d70d54694b7995d03c8c607ae706` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.435737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b8086f-dcd5-46e1-8be0-f632ac5c3a57 admin admin] RBAC: Authorizing `identity:delete_project(project_id=af86d70d54694b7995d03c8c607ae706)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:59.438505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b8086f-dcd5-46e1-8be0-f632ac5c3a57 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:59.444860 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c8e1b2-bd17-4367-b6c6-0842e4029f7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:59.446325 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c8e1b2-bd17-4367-b6c6-0842e4029f7d admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.446813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c8e1b2-bd17-4367-b6c6-0842e4029f7d admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.448088 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c8e1b2-bd17-4367-b6c6-0842e4029f7d admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.448960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c8e1b2-bd17-4367-b6c6-0842e4029f7d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:57:59.449483 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c8e1b2-bd17-4367-b6c6-0842e4029f7d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:59.452048 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c8e1b2-bd17-4367-b6c6-0842e4029f7d admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:59.478972 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 872/1721] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:57:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:57:59.484204 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c8e1b2-bd17-4367-b6c6-0842e4029f7d admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:59.496226 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c8e1b2-bd17-4367-b6c6-0842e4029f7d admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:59.504488 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-60b8086f-dcd5-46e1-8be0-f632ac5c3a57 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'af86d70d54694b7995d03c8c607ae706', 'request_id': 'req-60b8086f-dcd5-46e1-8be0-f632ac5c3a57'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:59.516390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:59.516652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-60b8086f-dcd5-46e1-8be0-f632ac5c3a57 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'af86d70d54694b7995d03c8c607ae706', 'request_id': 'req-60b8086f-dcd5-46e1-8be0-f632ac5c3a57'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:59.525648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-60b8086f-dcd5-46e1-8be0-f632ac5c3a57 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'af86d70d54694b7995d03c8c607ae706', 'request_id': 'req-60b8086f-dcd5-46e1-8be0-f632ac5c3a57'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:59.544400 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-60b8086f-dcd5-46e1-8be0-f632ac5c3a57 admin admin] The token cache is being invalidate because project af86d70d54694b7995d03c8c607ae706 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:57:59.544707 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-60b8086f-dcd5-46e1-8be0-f632ac5c3a57 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:59.545973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-60b8086f-dcd5-46e1-8be0-f632ac5c3a57 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:57:59.547213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 850/1722] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:57:59 2022] DELETE /identity/v3/projects/af86d70d54694b7995d03c8c607ae706 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:59.552510 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8c1520-9114-490d-92a3-5306e549027c None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.553147 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8c1520-9114-490d-92a3-5306e549027c None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.553270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8c1520-9114-490d-92a3-5306e549027c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.578490 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-4c8c1520-9114-490d-92a3-5306e549027c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:59.579767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-4c8c1520-9114-490d-92a3-5306e549027c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:59.599642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9b7bd3-51cc-45e4-8a84-129a71f484a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:59.600296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9b7bd3-51cc-45e4-8a84-129a71f484a2 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.600574 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9b7bd3-51cc-45e4-8a84-129a71f484a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.600846 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9b7bd3-51cc-45e4-8a84-129a71f484a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.601474 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9b7bd3-51cc-45e4-8a84-129a71f484a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:59.603416 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9b7bd3-51cc-45e4-8a84-129a71f484a2 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:59.639860 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 873/1723] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:59 2022] POST /identity/v3/projects => generated 361 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:57:59.644524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9b7bd3-51cc-45e4-8a84-129a71f484a2 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:59.649132 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8e9b7bd3-51cc-45e4-8a84-129a71f484a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:59.667590 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 851/1724] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:57:59.673332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b646bcd-0f5e-4f00-87e3-2a29d5652f70 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.673618 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b646bcd-0f5e-4f00-87e3-2a29d5652f70 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.673887 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b646bcd-0f5e-4f00-87e3-2a29d5652f70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.695205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-6b646bcd-0f5e-4f00-87e3-2a29d5652f70 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:59.696666 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6b646bcd-0f5e-4f00-87e3-2a29d5652f70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:57:59.717022 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:57:59.731586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9b7bd3-51cc-45e4-8a84-129a71f484a2 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:59.740025 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a9fe32-8be4-4071-848c-0239462a3c36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:59.740985 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a9fe32-8be4-4071-848c-0239462a3c36 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.741284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a9fe32-8be4-4071-848c-0239462a3c36 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.741645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a9fe32-8be4-4071-848c-0239462a3c36 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.742289 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a9fe32-8be4-4071-848c-0239462a3c36 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:59.746211 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a9fe32-8be4-4071-848c-0239462a3c36 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:59.772686 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 852/1725] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:57:59.773171 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 874/1726] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:59 2022] POST /identity/v3/users => generated 281 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:57:59.778135 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b646bcd-0f5e-4f00-87e3-2a29d5652f70 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:59.779794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a9fe32-8be4-4071-848c-0239462a3c36 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:59.787651 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b646bcd-0f5e-4f00-87e3-2a29d5652f70 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:59.791954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a9fe32-8be4-4071-848c-0239462a3c36 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:59.793194 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed340456-2f9a-4490-8e2e-b18ece1690b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:59.793794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed340456-2f9a-4490-8e2e-b18ece1690b7 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.794075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed340456-2f9a-4490-8e2e-b18ece1690b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.794347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed340456-2f9a-4490-8e2e-b18ece1690b7 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.795039 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed340456-2f9a-4490-8e2e-b18ece1690b7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:57:59.795342 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed340456-2f9a-4490-8e2e-b18ece1690b7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:59.796970 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed340456-2f9a-4490-8e2e-b18ece1690b7 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:59.797226 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dd6bfa-e107-44c0-bd43-8c965f6d5393 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:59.800321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dd6bfa-e107-44c0-bd43-8c965f6d5393 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.800609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dd6bfa-e107-44c0-bd43-8c965f6d5393 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.800881 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dd6bfa-e107-44c0-bd43-8c965f6d5393 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.801530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4dd6bfa-e107-44c0-bd43-8c965f6d5393 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:59.803244 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4dd6bfa-e107-44c0-bd43-8c965f6d5393 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:59.809646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 853/1727] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:57:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:57:59.813854 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 875/1728] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:59 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:59.814048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed340456-2f9a-4490-8e2e-b18ece1690b7 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:59.817853 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dd6bfa-e107-44c0-bd43-8c965f6d5393 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:59.825234 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed340456-2f9a-4490-8e2e-b18ece1690b7 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:59.830088 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d4be5e-e0aa-49eb-9b4f-744290a43019 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:59.830220 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dd6bfa-e107-44c0-bd43-8c965f6d5393 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:59.830769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d4be5e-e0aa-49eb-9b4f-744290a43019 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.831165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d4be5e-e0aa-49eb-9b4f-744290a43019 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.831584 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d4be5e-e0aa-49eb-9b4f-744290a43019 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.832567 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d4be5e-e0aa-49eb-9b4f-744290a43019 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:59.834579 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d4be5e-e0aa-49eb-9b4f-744290a43019 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:59.835234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3ee25a-380e-472a-9c7d-b10ba0b28f46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:59.835874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3ee25a-380e-472a-9c7d-b10ba0b28f46 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.836151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3ee25a-380e-472a-9c7d-b10ba0b28f46 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.836948 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3ee25a-380e-472a-9c7d-b10ba0b28f46 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.837982 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3ee25a-380e-472a-9c7d-b10ba0b28f46 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:59.839851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3ee25a-380e-472a-9c7d-b10ba0b28f46 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:59.855769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 876/1729] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:57:59 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:57:59.857498 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3ee25a-380e-472a-9c7d-b10ba0b28f46 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:59.869113 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3ee25a-380e-472a-9c7d-b10ba0b28f46 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:59.873299 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 854/1730] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:57:59 2022] POST /identity/v3/projects => generated 375 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:57:59.874445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e32a0e-05d0-48e9-80e1-a4c7da4b6cd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:57:59.875097 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e32a0e-05d0-48e9-80e1-a4c7da4b6cd9 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.875374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e32a0e-05d0-48e9-80e1-a4c7da4b6cd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.875778 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e32a0e-05d0-48e9-80e1-a4c7da4b6cd9 admin admin] PATH_INFO: `/v3/projects/3146272b16d74a0bb6957ace1947fbcb/users/1133069187d54d3f93b279c5792597d2/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.877549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d4be5e-e0aa-49eb-9b4f-744290a43019 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:57:59.892950 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e32a0e-05d0-48e9-80e1-a4c7da4b6cd9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3146272b16d74a0bb6957ace1947fbcb, user_id=1133069187d54d3f93b279c5792597d2, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:57:59.895584 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e32a0e-05d0-48e9-80e1-a4c7da4b6cd9 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:57:59.916818 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 877/1731] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:57:59 2022] PUT /identity/v3/projects/3146272b16d74a0bb6957ace1947fbcb/users/1133069187d54d3f93b279c5792597d2/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:57:59.928364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89606d87-064c-4ed5-b89b-0a6df5506aee None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:57:59.928830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89606d87-064c-4ed5-b89b-0a6df5506aee None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:57:59.929111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89606d87-064c-4ed5-b89b-0a6df5506aee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:57:59.970339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d4be5e-e0aa-49eb-9b4f-744290a43019 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:57:59.970617 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-89606d87-064c-4ed5-b89b-0a6df5506aee None None] MFA Rules not processed for user `1133069187d54d3f93b279c5792597d2`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:57:59.971922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-89606d87-064c-4ed5-b89b-0a6df5506aee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:00.025682 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:00.065519 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c1eb3c-ebc2-4b6d-9e8c-501dad3f505f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:00.066260 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 878/1732] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:57:59 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:00.066487 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c1eb3c-ebc2-4b6d-9e8c-501dad3f505f admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:00.066870 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c1eb3c-ebc2-4b6d-9e8c-501dad3f505f admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:00.067147 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c1eb3c-ebc2-4b6d-9e8c-501dad3f505f admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:00.067825 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c1eb3c-ebc2-4b6d-9e8c-501dad3f505f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:00.069768 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c1eb3c-ebc2-4b6d-9e8c-501dad3f505f admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:00.099384 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 855/1733] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:57:59 2022] POST /identity/v3/users => generated 288 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:00.105004 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89606d87-064c-4ed5-b89b-0a6df5506aee None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:00.112278 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c1eb3c-ebc2-4b6d-9e8c-501dad3f505f admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:00.112278 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-97c1eb3c-ebc2-4b6d-9e8c-501dad3f505f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:00.120329 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89606d87-064c-4ed5-b89b-0a6df5506aee None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:00.125202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16550ae-2e19-4c0e-a9b8-8c9c151173be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:00.125800 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16550ae-2e19-4c0e-a9b8-8c9c151173be admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:00.126080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16550ae-2e19-4c0e-a9b8-8c9c151173be admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:00.126349 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16550ae-2e19-4c0e-a9b8-8c9c151173be admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:00.127096 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16550ae-2e19-4c0e-a9b8-8c9c151173be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:00.128814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16550ae-2e19-4c0e-a9b8-8c9c151173be admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:00.142855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 879/1734] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:00 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:00.142855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16550ae-2e19-4c0e-a9b8-8c9c151173be admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:00.144776 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-c16550ae-2e19-4c0e-a9b8-8c9c151173be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:00.146729 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:00.175847 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c1eb3c-ebc2-4b6d-9e8c-501dad3f505f admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:00.179827 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcb03d0-81b2-4658-9195-dd8e29c4a7f5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:00.180502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb03d0-81b2-4658-9195-dd8e29c4a7f5 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:00.180982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb03d0-81b2-4658-9195-dd8e29c4a7f5 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:00.181266 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb03d0-81b2-4658-9195-dd8e29c4a7f5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:00.190088 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcb03d0-81b2-4658-9195-dd8e29c4a7f5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:00.191567 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcb03d0-81b2-4658-9195-dd8e29c4a7f5 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:00.200922 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 856/1735] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:00 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:00.201735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcb03d0-81b2-4658-9195-dd8e29c4a7f5 service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:00.205517 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16550ae-2e19-4c0e-a9b8-8c9c151173be admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:00.215096 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcb03d0-81b2-4658-9195-dd8e29c4a7f5 service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:00.216254 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1519bf-adb3-4889-8895-51cee63c6eea service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:00.216867 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1519bf-adb3-4889-8895-51cee63c6eea service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:00.217464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1519bf-adb3-4889-8895-51cee63c6eea service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:00.217746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1519bf-adb3-4889-8895-51cee63c6eea service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:00.219518 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8c1519bf-adb3-4889-8895-51cee63c6eea service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:00.220163 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccc5455-e4f3-4704-b71e-c3e64ee7966f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:00.220752 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccc5455-e4f3-4704-b71e-c3e64ee7966f admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:00.221028 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccc5455-e4f3-4704-b71e-c3e64ee7966f admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:00.221297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccc5455-e4f3-4704-b71e-c3e64ee7966f admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:00.221925 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ccc5455-e4f3-4704-b71e-c3e64ee7966f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:00.223687 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ccc5455-e4f3-4704-b71e-c3e64ee7966f admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:00.242032 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 857/1736] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:00 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:00.242032 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccc5455-e4f3-4704-b71e-c3e64ee7966f admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:00.249785 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccc5455-e4f3-4704-b71e-c3e64ee7966f admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:00.254403 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe076df-ddc2-492f-a666-1158dfa348f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:00.255009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe076df-ddc2-492f-a666-1158dfa348f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:00.255287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe076df-ddc2-492f-a666-1158dfa348f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:00.255579 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe076df-ddc2-492f-a666-1158dfa348f1 admin admin] PATH_INFO: `/v3/users/1133069187d54d3f93b279c5792597d2` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:00.256695 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe076df-ddc2-492f-a666-1158dfa348f1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1133069187d54d3f93b279c5792597d2)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:00.259140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe076df-ddc2-492f-a666-1158dfa348f1 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:00.301496 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1519bf-adb3-4889-8895-51cee63c6eea service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:00.303806 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1519bf-adb3-4889-8895-51cee63c6eea service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:00.307768 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-afe076df-ddc2-492f-a666-1158dfa348f1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1133069187d54d3f93b279c5792597d2'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:00.313793 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 880/1737] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:00 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:00.314570 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1519bf-adb3-4889-8895-51cee63c6eea service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:00.314815 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-afe076df-ddc2-492f-a666-1158dfa348f1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1133069187d54d3f93b279c5792597d2'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:00.315651 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-afe076df-ddc2-492f-a666-1158dfa348f1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1133069187d54d3f93b279c5792597d2'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:00.316268 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-afe076df-ddc2-492f-a666-1158dfa348f1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1133069187d54d3f93b279c5792597d2'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:00.324905 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1519bf-adb3-4889-8895-51cee63c6eea service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:00.329841 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e403ba-1ebe-4933-a127-f6cd2c33f5e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:00.330571 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e403ba-1ebe-4933-a127-f6cd2c33f5e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:00.330768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e403ba-1ebe-4933-a127-f6cd2c33f5e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:00.331040 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e403ba-1ebe-4933-a127-f6cd2c33f5e1 admin admin] PATH_INFO: `/v3/projects/6d8b9979a32d4f019261033e05440917/users/c4ef2932d19241abb23fabeff26f4617/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:00.331829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-afe076df-ddc2-492f-a666-1158dfa348f1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1133069187d54d3f93b279c5792597d2'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:00.348569 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 858/1738] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:00 2022] DELETE /identity/v3/users/1133069187d54d3f93b279c5792597d2 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:00.348843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e403ba-1ebe-4933-a127-f6cd2c33f5e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d8b9979a32d4f019261033e05440917, user_id=c4ef2932d19241abb23fabeff26f4617, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:00.351544 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e403ba-1ebe-4933-a127-f6cd2c33f5e1 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:00.353456 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafaa36a-8f96-4376-861a-4c5130cc74ca None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:00.353749 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafaa36a-8f96-4376-861a-4c5130cc74ca None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:00.354032 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafaa36a-8f96-4376-861a-4c5130cc74ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:00.367179 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 881/1739] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:00 2022] PUT /identity/v3/projects/6d8b9979a32d4f019261033e05440917/users/c4ef2932d19241abb23fabeff26f4617/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:00.376790 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-eafaa36a-8f96-4376-861a-4c5130cc74ca None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:00.378086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-eafaa36a-8f96-4376-861a-4c5130cc74ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:00.379654 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589d5f02-a1ec-4f80-95b2-3cd776405950 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:00.379982 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589d5f02-a1ec-4f80-95b2-3cd776405950 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:00.380325 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589d5f02-a1ec-4f80-95b2-3cd776405950 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:00.412824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-589d5f02-a1ec-4f80-95b2-3cd776405950 None None] MFA Rules not processed for user `c4ef2932d19241abb23fabeff26f4617`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:00.414144 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-589d5f02-a1ec-4f80-95b2-3cd776405950 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:00.447023 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 859/1740] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:00.456726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafaa36a-8f96-4376-861a-4c5130cc74ca None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:00.458336 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-eafaa36a-8f96-4376-861a-4c5130cc74ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:00.459013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:00.489471 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 882/1741] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:00 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:00.505716 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:00.505854 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589d5f02-a1ec-4f80-95b2-3cd776405950 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:00.505854 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-589d5f02-a1ec-4f80-95b2-3cd776405950 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:00.544728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafaa36a-8f96-4376-861a-4c5130cc74ca None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:00.549015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512f21f2-3408-4fb1-9e58-43b0616aefdd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:00.549595 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512f21f2-3408-4fb1-9e58-43b0616aefdd service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:00.549880 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512f21f2-3408-4fb1-9e58-43b0616aefdd service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:00.550147 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512f21f2-3408-4fb1-9e58-43b0616aefdd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:00.561985 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512f21f2-3408-4fb1-9e58-43b0616aefdd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:00.563426 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512f21f2-3408-4fb1-9e58-43b0616aefdd service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:00.574503 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 860/1742] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:58:00.603977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589d5f02-a1ec-4f80-95b2-3cd776405950 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:00.611136 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a188b71-a6b2-41ea-9f80-02516a85f193 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:00.611865 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a188b71-a6b2-41ea-9f80-02516a85f193 service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:00.612143 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a188b71-a6b2-41ea-9f80-02516a85f193 service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:00.612451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a188b71-a6b2-41ea-9f80-02516a85f193 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:00.622216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a188b71-a6b2-41ea-9f80-02516a85f193 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:00.623989 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a188b71-a6b2-41ea-9f80-02516a85f193 service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:00.634882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 883/1743] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:00 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:00.919405 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512f21f2-3408-4fb1-9e58-43b0616aefdd service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:00.921181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-512f21f2-3408-4fb1-9e58-43b0616aefdd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:00.936146 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512f21f2-3408-4fb1-9e58-43b0616aefdd service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:00.937524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a188b71-a6b2-41ea-9f80-02516a85f193 service nova] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:00.939197 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-4a188b71-a6b2-41ea-9f80-02516a85f193 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:00.941453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cff4a5-3d61-4b2a-8ffe-cae51e18c3f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:00.942168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cff4a5-3d61-4b2a-8ffe-cae51e18c3f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:00.942493 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cff4a5-3d61-4b2a-8ffe-cae51e18c3f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:00.942836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cff4a5-3d61-4b2a-8ffe-cae51e18c3f2 admin admin] PATH_INFO: `/v3/users/c4ef2932d19241abb23fabeff26f4617` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:00.944135 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12cff4a5-3d61-4b2a-8ffe-cae51e18c3f2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c4ef2932d19241abb23fabeff26f4617)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:00.946543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12cff4a5-3d61-4b2a-8ffe-cae51e18c3f2 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:00.956365 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a188b71-a6b2-41ea-9f80-02516a85f193 service nova] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:00.963212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e104ef-636a-4288-b336-4c7856952df6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:00.963877 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e104ef-636a-4288-b336-4c7856952df6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:00.964207 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e104ef-636a-4288-b336-4c7856952df6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:00.964458 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e104ef-636a-4288-b336-4c7856952df6 admin admin] PATH_INFO: `/v3/projects/3146272b16d74a0bb6957ace1947fbcb` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:00.965633 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e104ef-636a-4288-b336-4c7856952df6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3146272b16d74a0bb6957ace1947fbcb)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:00.967513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e104ef-636a-4288-b336-4c7856952df6 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:01.006585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-12cff4a5-3d61-4b2a-8ffe-cae51e18c3f2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c4ef2932d19241abb23fabeff26f4617'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:01.014037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-12cff4a5-3d61-4b2a-8ffe-cae51e18c3f2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c4ef2932d19241abb23fabeff26f4617'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:01.014715 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-12cff4a5-3d61-4b2a-8ffe-cae51e18c3f2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c4ef2932d19241abb23fabeff26f4617'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:01.015329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-12cff4a5-3d61-4b2a-8ffe-cae51e18c3f2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c4ef2932d19241abb23fabeff26f4617'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:01.021486 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-47e104ef-636a-4288-b336-4c7856952df6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3146272b16d74a0bb6957ace1947fbcb', 'request_id': 'req-47e104ef-636a-4288-b336-4c7856952df6'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:01.026939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-47e104ef-636a-4288-b336-4c7856952df6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3146272b16d74a0bb6957ace1947fbcb', 'request_id': 'req-47e104ef-636a-4288-b336-4c7856952df6'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:01.028638 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-12cff4a5-3d61-4b2a-8ffe-cae51e18c3f2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c4ef2932d19241abb23fabeff26f4617'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:01.033518 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-47e104ef-636a-4288-b336-4c7856952df6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3146272b16d74a0bb6957ace1947fbcb', 'request_id': 'req-47e104ef-636a-4288-b336-4c7856952df6'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:01.043883 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 861/1744] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:00 2022] DELETE /identity/v3/users/c4ef2932d19241abb23fabeff26f4617 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:01.047150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-47e104ef-636a-4288-b336-4c7856952df6 admin admin] The token cache is being invalidate because project 3146272b16d74a0bb6957ace1947fbcb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:01.047449 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-47e104ef-636a-4288-b336-4c7856952df6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:01.048115 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-47e104ef-636a-4288-b336-4c7856952df6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:01.048585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa12fee-4a45-4354-a78a-b89f6238d94a None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:01.048870 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa12fee-4a45-4354-a78a-b89f6238d94a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:01.049143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa12fee-4a45-4354-a78a-b89f6238d94a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:01.049590 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 884/1745] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:00 2022] DELETE /identity/v3/projects/3146272b16d74a0bb6957ace1947fbcb => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:58:01.054086 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b9cf24-7562-42d0-8ce5-30a6ec1d97b4 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:01.054386 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b9cf24-7562-42d0-8ce5-30a6ec1d97b4 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:01.054648 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b9cf24-7562-42d0-8ce5-30a6ec1d97b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:01.071037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-baa12fee-4a45-4354-a78a-b89f6238d94a None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:01.072828 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-baa12fee-4a45-4354-a78a-b89f6238d94a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:01.074212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-02b9cf24-7562-42d0-8ce5-30a6ec1d97b4 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:01.075372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-02b9cf24-7562-42d0-8ce5-30a6ec1d97b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:01.121687 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:01.136915 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 862/1746] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:01.143283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 885/1747] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:01.148010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa12fee-4a45-4354-a78a-b89f6238d94a None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:01.150434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-baa12fee-4a45-4354-a78a-b89f6238d94a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:01.151150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd8bf14-f35d-4ce0-b1f3-3a350a1a48f8 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:01.151519 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd8bf14-f35d-4ce0-b1f3-3a350a1a48f8 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:01.151740 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd8bf14-f35d-4ce0-b1f3-3a350a1a48f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:01.172658 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-0cd8bf14-f35d-4ce0-b1f3-3a350a1a48f8 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:01.174032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0cd8bf14-f35d-4ce0-b1f3-3a350a1a48f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:01.184331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 886/1748] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:01.188531 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd8bf14-f35d-4ce0-b1f3-3a350a1a48f8 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:01.198784 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd8bf14-f35d-4ce0-b1f3-3a350a1a48f8 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:01.204270 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e935ab8-6f07-48c3-88bc-48bc20f09585 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:01.204884 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e935ab8-6f07-48c3-88bc-48bc20f09585 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:01.205159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e935ab8-6f07-48c3-88bc-48bc20f09585 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:01.205432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e935ab8-6f07-48c3-88bc-48bc20f09585 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:01.206154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e935ab8-6f07-48c3-88bc-48bc20f09585 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:58:01.206373 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e935ab8-6f07-48c3-88bc-48bc20f09585 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:01.208669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e935ab8-6f07-48c3-88bc-48bc20f09585 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:01.214366 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:01.220668 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa12fee-4a45-4354-a78a-b89f6238d94a None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:01.225599 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49df6cd7-3dc0-4b9a-b2d7-8cf1200bf9aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:01.226335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49df6cd7-3dc0-4b9a-b2d7-8cf1200bf9aa service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:01.227363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 887/1749] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:58:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:58:01.227670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49df6cd7-3dc0-4b9a-b2d7-8cf1200bf9aa service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:01.227966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49df6cd7-3dc0-4b9a-b2d7-8cf1200bf9aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:01.231738 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e935ab8-6f07-48c3-88bc-48bc20f09585 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:01.238111 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49df6cd7-3dc0-4b9a-b2d7-8cf1200bf9aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:01.239573 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49df6cd7-3dc0-4b9a-b2d7-8cf1200bf9aa service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:01.248362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e935ab8-6f07-48c3-88bc-48bc20f09585 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:01.250844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 863/1750] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:01.253699 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0233b5d5-f92f-420d-b77c-5d3ed1f17601 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:01.254347 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0233b5d5-f92f-420d-b77c-5d3ed1f17601 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:01.254629 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0233b5d5-f92f-420d-b77c-5d3ed1f17601 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:01.254934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0233b5d5-f92f-420d-b77c-5d3ed1f17601 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:01.255606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0233b5d5-f92f-420d-b77c-5d3ed1f17601 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:01.257569 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0233b5d5-f92f-420d-b77c-5d3ed1f17601 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:01.294007 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 888/1751] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:58:01 2022] POST /identity/v3/projects => generated 373 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:58:01.298031 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49df6cd7-3dc0-4b9a-b2d7-8cf1200bf9aa service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:01.309782 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49df6cd7-3dc0-4b9a-b2d7-8cf1200bf9aa service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:01.375696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e7347a-a0f4-4a8e-8414-7eb7adb7f965 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:01.376341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e7347a-a0f4-4a8e-8414-7eb7adb7f965 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:01.376759 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e7347a-a0f4-4a8e-8414-7eb7adb7f965 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:01.377046 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e7347a-a0f4-4a8e-8414-7eb7adb7f965 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:01.377701 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e7347a-a0f4-4a8e-8414-7eb7adb7f965 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:01.379769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e7347a-a0f4-4a8e-8414-7eb7adb7f965 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:01.402105 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 864/1752] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:58:01 2022] POST /identity/v3/users => generated 287 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:01.406082 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0233b5d5-f92f-420d-b77c-5d3ed1f17601 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:01.417301 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0233b5d5-f92f-420d-b77c-5d3ed1f17601 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:01.422352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2447fcc0-1575-43b3-bdbb-45b14b2a41df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:01.422993 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2447fcc0-1575-43b3-bdbb-45b14b2a41df admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:01.423304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2447fcc0-1575-43b3-bdbb-45b14b2a41df admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:01.423602 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2447fcc0-1575-43b3-bdbb-45b14b2a41df admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:01.424244 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2447fcc0-1575-43b3-bdbb-45b14b2a41df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:01.428447 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2447fcc0-1575-43b3-bdbb-45b14b2a41df admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:01.440843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 889/1753] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:01 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:01.445606 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e7347a-a0f4-4a8e-8414-7eb7adb7f965 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:01.456844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e7347a-a0f4-4a8e-8414-7eb7adb7f965 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:01.463664 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9242c535-3670-4dd2-b00e-6699ea24568d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:01.464272 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9242c535-3670-4dd2-b00e-6699ea24568d admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:01.464550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9242c535-3670-4dd2-b00e-6699ea24568d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:01.464820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9242c535-3670-4dd2-b00e-6699ea24568d admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:01.465493 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9242c535-3670-4dd2-b00e-6699ea24568d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:01.467109 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9242c535-3670-4dd2-b00e-6699ea24568d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:01.482628 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2447fcc0-1575-43b3-bdbb-45b14b2a41df admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:01.483757 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 865/1754] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:01 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:01.494728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2447fcc0-1575-43b3-bdbb-45b14b2a41df admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:01.499874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bd0ee2-5709-4a0f-83df-9cd715a14a39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:01.500860 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bd0ee2-5709-4a0f-83df-9cd715a14a39 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:01.501148 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bd0ee2-5709-4a0f-83df-9cd715a14a39 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:01.501420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bd0ee2-5709-4a0f-83df-9cd715a14a39 admin admin] PATH_INFO: `/v3/projects/4075982a0ef04a6a8edbe37fde6cb657/users/dd1d89a64c274cefb03a68c3233cfa51/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:01.520608 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bd0ee2-5709-4a0f-83df-9cd715a14a39 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4075982a0ef04a6a8edbe37fde6cb657, user_id=dd1d89a64c274cefb03a68c3233cfa51, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:01.524737 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bd0ee2-5709-4a0f-83df-9cd715a14a39 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:01.544478 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 890/1755] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:01 2022] PUT /identity/v3/projects/4075982a0ef04a6a8edbe37fde6cb657/users/dd1d89a64c274cefb03a68c3233cfa51/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:01.549711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b3c950-e230-4951-8a6c-3893a048182e None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:01.549999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b3c950-e230-4951-8a6c-3893a048182e None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:01.550278 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b3c950-e230-4951-8a6c-3893a048182e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:01.577920 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-47b3c950-e230-4951-8a6c-3893a048182e None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:01.579083 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-47b3c950-e230-4951-8a6c-3893a048182e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:01.584787 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bd0ee2-5709-4a0f-83df-9cd715a14a39 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:01.587505 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-57bd0ee2-5709-4a0f-83df-9cd715a14a39 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:01.667342 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 866/1756] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:01.679195 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b3c950-e230-4951-8a6c-3893a048182e None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:01.682033 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bd0ee2-5709-4a0f-83df-9cd715a14a39 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:01.688264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360e93ed-962a-4b4b-90ea-0edcff66a151 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:01.689111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360e93ed-962a-4b4b-90ea-0edcff66a151 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:01.689478 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360e93ed-962a-4b4b-90ea-0edcff66a151 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:01.689825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360e93ed-962a-4b4b-90ea-0edcff66a151 admin admin] PATH_INFO: `/v3/projects/6d8b9979a32d4f019261033e05440917` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:01.691078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360e93ed-962a-4b4b-90ea-0edcff66a151 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d8b9979a32d4f019261033e05440917)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:01.693081 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360e93ed-962a-4b4b-90ea-0edcff66a151 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:01.746280 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-360e93ed-962a-4b4b-90ea-0edcff66a151 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d8b9979a32d4f019261033e05440917', 'request_id': 'req-360e93ed-962a-4b4b-90ea-0edcff66a151'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:01.749040 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b3c950-e230-4951-8a6c-3893a048182e None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:01.753445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-360e93ed-962a-4b4b-90ea-0edcff66a151 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d8b9979a32d4f019261033e05440917', 'request_id': 'req-360e93ed-962a-4b4b-90ea-0edcff66a151'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:01.760969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-360e93ed-962a-4b4b-90ea-0edcff66a151 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d8b9979a32d4f019261033e05440917', 'request_id': 'req-360e93ed-962a-4b4b-90ea-0edcff66a151'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:01.778533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-360e93ed-962a-4b4b-90ea-0edcff66a151 admin admin] The token cache is being invalidate because project 6d8b9979a32d4f019261033e05440917 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:01.778989 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-360e93ed-962a-4b4b-90ea-0edcff66a151 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:01.779744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-360e93ed-962a-4b4b-90ea-0edcff66a151 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:01.784512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 891/1757] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:01 2022] DELETE /identity/v3/projects/6d8b9979a32d4f019261033e05440917 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:58:01.784959 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:01.785990 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27556978-66f3-40e6-98e3-091782c250ee None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:01.786262 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27556978-66f3-40e6-98e3-091782c250ee None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:01.786524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27556978-66f3-40e6-98e3-091782c250ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:01.799390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:01.815087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-27556978-66f3-40e6-98e3-091782c250ee None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:01.817423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-27556978-66f3-40e6-98e3-091782c250ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:01.839060 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4633f0cb-f46e-4eed-996e-d0cc1218dfc8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:01.840740 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4633f0cb-f46e-4eed-996e-d0cc1218dfc8 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:01.840740 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4633f0cb-f46e-4eed-996e-d0cc1218dfc8 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:01.840879 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4633f0cb-f46e-4eed-996e-d0cc1218dfc8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:01.843453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-4633f0cb-f46e-4eed-996e-d0cc1218dfc8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:01.897383 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 892/1758] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:01.904711 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02c52ec-d8ca-4393-9093-81137f9f3721 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:01.904935 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02c52ec-d8ca-4393-9093-81137f9f3721 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:01.905357 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02c52ec-d8ca-4393-9093-81137f9f3721 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:01.925112 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4633f0cb-f46e-4eed-996e-d0cc1218dfc8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:01.926546 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4633f0cb-f46e-4eed-996e-d0cc1218dfc8 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:01.928175 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-c02c52ec-d8ca-4393-9093-81137f9f3721 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:01.929482 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-c02c52ec-d8ca-4393-9093-81137f9f3721 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:01.937918 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 867/1759] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:58:01.938398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 893/1760] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:01.942622 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4633f0cb-f46e-4eed-996e-d0cc1218dfc8 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:01.959735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4633f0cb-f46e-4eed-996e-d0cc1218dfc8 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:01.959735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9132e5b1-f77f-4714-8aaf-f694a6ef8a8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:01.959735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9132e5b1-f77f-4714-8aaf-f694a6ef8a8b admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:01.959735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9132e5b1-f77f-4714-8aaf-f694a6ef8a8b admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:01.960077 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9132e5b1-f77f-4714-8aaf-f694a6ef8a8b admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:01.960077 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9132e5b1-f77f-4714-8aaf-f694a6ef8a8b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:58:01.960077 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9132e5b1-f77f-4714-8aaf-f694a6ef8a8b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:01.961514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9132e5b1-f77f-4714-8aaf-f694a6ef8a8b admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:01.974385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 868/1761] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:58:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:58:01.979410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02c52ec-d8ca-4393-9093-81137f9f3721 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:01.989085 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02c52ec-d8ca-4393-9093-81137f9f3721 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:01.994782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c62e380-17c0-45e6-a6d9-11b248a1e495 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:01.995393 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c62e380-17c0-45e6-a6d9-11b248a1e495 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:01.995706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c62e380-17c0-45e6-a6d9-11b248a1e495 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:01.995979 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c62e380-17c0-45e6-a6d9-11b248a1e495 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:01.996621 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c62e380-17c0-45e6-a6d9-11b248a1e495 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:01.998539 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c62e380-17c0-45e6-a6d9-11b248a1e495 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:02.030234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 894/1762] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:58:01 2022] POST /identity/v3/projects => generated 373 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:02.034897 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9132e5b1-f77f-4714-8aaf-f694a6ef8a8b admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:02.054015 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9132e5b1-f77f-4714-8aaf-f694a6ef8a8b admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:02.129824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e71e644-979c-496b-a35c-e62ed02e421f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:02.130820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e71e644-979c-496b-a35c-e62ed02e421f admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:02.131223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e71e644-979c-496b-a35c-e62ed02e421f admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:02.131932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e71e644-979c-496b-a35c-e62ed02e421f admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:02.132375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e71e644-979c-496b-a35c-e62ed02e421f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:02.135356 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e71e644-979c-496b-a35c-e62ed02e421f admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:02.169148 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 869/1763] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:58:02 2022] POST /identity/v3/users => generated 287 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:02.173478 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c62e380-17c0-45e6-a6d9-11b248a1e495 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:02.195950 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c62e380-17c0-45e6-a6d9-11b248a1e495 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:02.199655 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed2aed2-86c8-408a-a34f-bdf152bded68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:02.200471 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed2aed2-86c8-408a-a34f-bdf152bded68 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:02.200737 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed2aed2-86c8-408a-a34f-bdf152bded68 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:02.201101 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed2aed2-86c8-408a-a34f-bdf152bded68 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:02.201939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed2aed2-86c8-408a-a34f-bdf152bded68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:02.203928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed2aed2-86c8-408a-a34f-bdf152bded68 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:02.218833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 895/1764] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:02 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:02.222804 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e71e644-979c-496b-a35c-e62ed02e421f admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:02.234338 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e71e644-979c-496b-a35c-e62ed02e421f admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:02.239950 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bb78a3-05c4-426a-958f-db3f8d495644 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:02.240564 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bb78a3-05c4-426a-958f-db3f8d495644 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:02.240847 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bb78a3-05c4-426a-958f-db3f8d495644 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:02.241119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bb78a3-05c4-426a-958f-db3f8d495644 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:02.241901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bb78a3-05c4-426a-958f-db3f8d495644 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:02.244062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bb78a3-05c4-426a-958f-db3f8d495644 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:02.259470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 870/1765] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:02 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:02.261001 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed2aed2-86c8-408a-a34f-bdf152bded68 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:02.272364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed2aed2-86c8-408a-a34f-bdf152bded68 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:02.277857 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26865c9c-de4a-48a0-b0b4-82683665fa0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:02.278776 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26865c9c-de4a-48a0-b0b4-82683665fa0d admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:02.279236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26865c9c-de4a-48a0-b0b4-82683665fa0d admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:02.279606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26865c9c-de4a-48a0-b0b4-82683665fa0d admin admin] PATH_INFO: `/v3/projects/fed05b44b6c640acbee4c86f22860a25/users/2c2f5247254d4c14b40216487b68f93e/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:02.299307 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26865c9c-de4a-48a0-b0b4-82683665fa0d admin admin] RBAC: Authorizing `identity:create_grant(project_id=fed05b44b6c640acbee4c86f22860a25, user_id=2c2f5247254d4c14b40216487b68f93e, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:02.301763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26865c9c-de4a-48a0-b0b4-82683665fa0d admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:02.318494 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 896/1766] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:02 2022] PUT /identity/v3/projects/fed05b44b6c640acbee4c86f22860a25/users/2c2f5247254d4c14b40216487b68f93e/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:02.330600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0040e962-e569-453c-9177-02e144bba891 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:02.330873 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0040e962-e569-453c-9177-02e144bba891 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:02.331219 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0040e962-e569-453c-9177-02e144bba891 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:02.370836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-0040e962-e569-453c-9177-02e144bba891 None None] MFA Rules not processed for user `2c2f5247254d4c14b40216487b68f93e`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:02.372335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-0040e962-e569-453c-9177-02e144bba891 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:02.416967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:02.472538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 871/1767] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:02 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:02.479186 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26865c9c-de4a-48a0-b0b4-82683665fa0d admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:02.551379 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26865c9c-de4a-48a0-b0b4-82683665fa0d admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:02.565375 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a73181-0057-47a0-a44d-2555226080a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:02.565788 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a73181-0057-47a0-a44d-2555226080a5 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:02.566076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a73181-0057-47a0-a44d-2555226080a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:02.566355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a73181-0057-47a0-a44d-2555226080a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:02.567002 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a73181-0057-47a0-a44d-2555226080a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:02.568945 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a73181-0057-47a0-a44d-2555226080a5 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:02.601039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 897/1768] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:58:02 2022] POST /identity/v3/projects => generated 371 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:02.602999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0040e962-e569-453c-9177-02e144bba891 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:02.614283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0040e962-e569-453c-9177-02e144bba891 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:02.684726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cf808d-b7dc-416c-9b3f-436465d88808 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:02.684726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cf808d-b7dc-416c-9b3f-436465d88808 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:02.684726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cf808d-b7dc-416c-9b3f-436465d88808 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:02.684726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cf808d-b7dc-416c-9b3f-436465d88808 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:02.685306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cf808d-b7dc-416c-9b3f-436465d88808 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:02.685825 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cf808d-b7dc-416c-9b3f-436465d88808 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:02.686777 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bad4366-c359-474a-9462-0aa75f4c4dc6 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:02.687259 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bad4366-c359-474a-9462-0aa75f4c4dc6 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:02.687668 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bad4366-c359-474a-9462-0aa75f4c4dc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:02.710721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 872/1769] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:58:02 2022] POST /identity/v3/users => generated 292 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:58:02.715149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cf808d-b7dc-416c-9b3f-436465d88808 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:02.725774 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-1bad4366-c359-474a-9462-0aa75f4c4dc6 None None] MFA Rules not processed for user `dd1d89a64c274cefb03a68c3233cfa51`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:02.725991 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cf808d-b7dc-416c-9b3f-436465d88808 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:02.727539 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1bad4366-c359-474a-9462-0aa75f4c4dc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:02.732166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e4a0c5-5b87-49bc-9312-e6d72f6a27af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:02.732748 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e4a0c5-5b87-49bc-9312-e6d72f6a27af admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:02.733650 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e4a0c5-5b87-49bc-9312-e6d72f6a27af admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:02.733650 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e4a0c5-5b87-49bc-9312-e6d72f6a27af admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:02.733953 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e4a0c5-5b87-49bc-9312-e6d72f6a27af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:02.735817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e4a0c5-5b87-49bc-9312-e6d72f6a27af admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:02.744725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:02.751083 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 873/1770] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:02 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 14:58:02.752029 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e4a0c5-5b87-49bc-9312-e6d72f6a27af admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:02.763377 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e4a0c5-5b87-49bc-9312-e6d72f6a27af admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:02.768387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882787bd-1f5b-4e5f-bac5-2f81d3ac0cd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:02.769048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882787bd-1f5b-4e5f-bac5-2f81d3ac0cd7 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:02.769333 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882787bd-1f5b-4e5f-bac5-2f81d3ac0cd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:02.769609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882787bd-1f5b-4e5f-bac5-2f81d3ac0cd7 admin admin] PATH_INFO: `/v3/projects/9dd801ad6a7a43ef988331158da99875/users/8de06df94c7b47719ae1f8123ddb60d3/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:02.788365 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882787bd-1f5b-4e5f-bac5-2f81d3ac0cd7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9dd801ad6a7a43ef988331158da99875, user_id=8de06df94c7b47719ae1f8123ddb60d3, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:02.792524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882787bd-1f5b-4e5f-bac5-2f81d3ac0cd7 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:02.806787 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 874/1771] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:02 2022] PUT /identity/v3/projects/9dd801ad6a7a43ef988331158da99875/users/8de06df94c7b47719ae1f8123ddb60d3/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:02.810606 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882787bd-1f5b-4e5f-bac5-2f81d3ac0cd7 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:02.818271 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 898/1772] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:02 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:02.830362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bad4366-c359-474a-9462-0aa75f4c4dc6 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:02.833244 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1bad4366-c359-474a-9462-0aa75f4c4dc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:02.896211 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882787bd-1f5b-4e5f-bac5-2f81d3ac0cd7 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:02.903976 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ae3d86-bac9-42ee-bff0-90bf6efd33c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:02.904882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ae3d86-bac9-42ee-bff0-90bf6efd33c8 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:02.905218 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ae3d86-bac9-42ee-bff0-90bf6efd33c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:02.906162 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ae3d86-bac9-42ee-bff0-90bf6efd33c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:02.908009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ae3d86-bac9-42ee-bff0-90bf6efd33c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:02.909915 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ae3d86-bac9-42ee-bff0-90bf6efd33c8 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:02.910628 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bad4366-c359-474a-9462-0aa75f4c4dc6 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:02.926391 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 875/1773] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:02 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 14:58:02.932534 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ae3d86-bac9-42ee-bff0-90bf6efd33c8 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:02.947098 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ae3d86-bac9-42ee-bff0-90bf6efd33c8 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:02.956362 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb03251-e917-4f09-88a2-5c7767b301d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:02.958139 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb03251-e917-4f09-88a2-5c7767b301d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:02.959577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb03251-e917-4f09-88a2-5c7767b301d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:02.959577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb03251-e917-4f09-88a2-5c7767b301d3 admin admin] PATH_INFO: `/v3/projects/9dd801ad6a7a43ef988331158da99875/users/8de06df94c7b47719ae1f8123ddb60d3/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:02.964927 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb03251-e917-4f09-88a2-5c7767b301d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9dd801ad6a7a43ef988331158da99875, user_id=8de06df94c7b47719ae1f8123ddb60d3, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:02.968026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb03251-e917-4f09-88a2-5c7767b301d3 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:02.977483 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd22e41-a580-462e-8241-60dc408c6850 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:02.978478 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd22e41-a580-462e-8241-60dc408c6850 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:02.978893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd22e41-a580-462e-8241-60dc408c6850 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:02.979394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd22e41-a580-462e-8241-60dc408c6850 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:02.981447 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 876/1774] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:02 2022] PUT /identity/v3/projects/9dd801ad6a7a43ef988331158da99875/users/8de06df94c7b47719ae1f8123ddb60d3/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:02.993666 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb333c0-ae5c-41a6-8bda-914c5cbf4d8a None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:02.993953 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb333c0-ae5c-41a6-8bda-914c5cbf4d8a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:02.994232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb333c0-ae5c-41a6-8bda-914c5cbf4d8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:03.009812 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:03.026903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-6eb333c0-ae5c-41a6-8bda-914c5cbf4d8a None None] MFA Rules not processed for user `8de06df94c7b47719ae1f8123ddb60d3`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:03.028390 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6eb333c0-ae5c-41a6-8bda-914c5cbf4d8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:03.052616 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd22e41-a580-462e-8241-60dc408c6850 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:03.054148 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd22e41-a580-462e-8241-60dc408c6850 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:03.064870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 899/1775] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:02 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:58:03.107583 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 877/1776] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:02 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:03.142501 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd22e41-a580-462e-8241-60dc408c6850 service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:03.144089 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-dcd22e41-a580-462e-8241-60dc408c6850 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:03.157100 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd22e41-a580-462e-8241-60dc408c6850 service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:03.178346 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:03.216512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c63618-d77e-4219-b6cf-08afc461e989 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:03.217180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c63618-d77e-4219-b6cf-08afc461e989 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:03.217457 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c63618-d77e-4219-b6cf-08afc461e989 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:03.217759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c63618-d77e-4219-b6cf-08afc461e989 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:03.228968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c63618-d77e-4219-b6cf-08afc461e989 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:03.230421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c63618-d77e-4219-b6cf-08afc461e989 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:03.240488 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 900/1777] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:03 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:03.869000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb333c0-ae5c-41a6-8bda-914c5cbf4d8a None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:03.930965 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb333c0-ae5c-41a6-8bda-914c5cbf4d8a None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:03.935709 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2933bfd-fced-4ed7-9b9b-c4739240c32e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:03.936333 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2933bfd-fced-4ed7-9b9b-c4739240c32e admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:03.936608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2933bfd-fced-4ed7-9b9b-c4739240c32e admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:03.936881 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2933bfd-fced-4ed7-9b9b-c4739240c32e admin admin] PATH_INFO: `/v3/users/2c2f5247254d4c14b40216487b68f93e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:03.937999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2933bfd-fced-4ed7-9b9b-c4739240c32e admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c2f5247254d4c14b40216487b68f93e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:03.939949 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2933bfd-fced-4ed7-9b9b-c4739240c32e admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:03.987410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b2933bfd-fced-4ed7-9b9b-c4739240c32e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c2f5247254d4c14b40216487b68f93e'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:03.994998 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b2933bfd-fced-4ed7-9b9b-c4739240c32e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c2f5247254d4c14b40216487b68f93e'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:03.995705 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b2933bfd-fced-4ed7-9b9b-c4739240c32e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c2f5247254d4c14b40216487b68f93e'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:03.996310 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b2933bfd-fced-4ed7-9b9b-c4739240c32e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c2f5247254d4c14b40216487b68f93e'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:04.008038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b2933bfd-fced-4ed7-9b9b-c4739240c32e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c2f5247254d4c14b40216487b68f93e'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:04.022031 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 878/1778] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:03 2022] DELETE /identity/v3/users/2c2f5247254d4c14b40216487b68f93e => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:04.026479 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ced872b-da19-48f1-a81e-3876c051a044 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:04.026964 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ced872b-da19-48f1-a81e-3876c051a044 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:04.027054 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ced872b-da19-48f1-a81e-3876c051a044 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:04.046500 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-2ced872b-da19-48f1-a81e-3876c051a044 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:04.047994 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2ced872b-da19-48f1-a81e-3876c051a044 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:04.106118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 901/1779] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:04.115940 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2933bfd-fced-4ed7-9b9b-c4739240c32e admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:04.117941 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-b2933bfd-fced-4ed7-9b9b-c4739240c32e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:04.131315 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:04.177318 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2933bfd-fced-4ed7-9b9b-c4739240c32e admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:04.181581 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0c10eb-60cb-4194-92c8-c0f89b6029d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:04.182121 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0c10eb-60cb-4194-92c8-c0f89b6029d2 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:04.182403 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0c10eb-60cb-4194-92c8-c0f89b6029d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:04.182675 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0c10eb-60cb-4194-92c8-c0f89b6029d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:04.191664 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0c10eb-60cb-4194-92c8-c0f89b6029d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:04.193094 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0c10eb-60cb-4194-92c8-c0f89b6029d2 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:04.203076 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 879/1780] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:04.468404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ced872b-da19-48f1-a81e-3876c051a044 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:04.469863 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2ced872b-da19-48f1-a81e-3876c051a044 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:04.483325 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ced872b-da19-48f1-a81e-3876c051a044 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:04.488081 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc84ce8-6692-4b4c-8ce2-0ae6c00cb7ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:04.488821 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc84ce8-6692-4b4c-8ce2-0ae6c00cb7ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:04.489164 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc84ce8-6692-4b4c-8ce2-0ae6c00cb7ba admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:04.489504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc84ce8-6692-4b4c-8ce2-0ae6c00cb7ba admin admin] PATH_INFO: `/v3/projects/fed05b44b6c640acbee4c86f22860a25` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:04.490711 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc84ce8-6692-4b4c-8ce2-0ae6c00cb7ba admin admin] RBAC: Authorizing `identity:delete_project(project_id=fed05b44b6c640acbee4c86f22860a25)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:04.493446 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc84ce8-6692-4b4c-8ce2-0ae6c00cb7ba admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:04.557180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8bc84ce8-6692-4b4c-8ce2-0ae6c00cb7ba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fed05b44b6c640acbee4c86f22860a25', 'request_id': 'req-8bc84ce8-6692-4b4c-8ce2-0ae6c00cb7ba'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:04.565994 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8bc84ce8-6692-4b4c-8ce2-0ae6c00cb7ba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fed05b44b6c640acbee4c86f22860a25', 'request_id': 'req-8bc84ce8-6692-4b4c-8ce2-0ae6c00cb7ba'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:04.578051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8bc84ce8-6692-4b4c-8ce2-0ae6c00cb7ba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fed05b44b6c640acbee4c86f22860a25', 'request_id': 'req-8bc84ce8-6692-4b4c-8ce2-0ae6c00cb7ba'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:04.593257 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8bc84ce8-6692-4b4c-8ce2-0ae6c00cb7ba admin admin] The token cache is being invalidate because project fed05b44b6c640acbee4c86f22860a25 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:04.593553 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8bc84ce8-6692-4b4c-8ce2-0ae6c00cb7ba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:04.594549 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8bc84ce8-6692-4b4c-8ce2-0ae6c00cb7ba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:04.596119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 902/1781] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:04 2022] DELETE /identity/v3/projects/fed05b44b6c640acbee4c86f22860a25 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:58:04.600500 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0c10eb-60cb-4194-92c8-c0f89b6029d2 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:04.600953 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0d4374-4a93-4726-b42d-cabf14852737 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:04.601373 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0d4374-4a93-4726-b42d-cabf14852737 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:04.602331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0d4374-4a93-4726-b42d-cabf14852737 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:04.612774 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0c10eb-60cb-4194-92c8-c0f89b6029d2 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:04.625196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-8e0d4374-4a93-4726-b42d-cabf14852737 None None] MFA Rules not processed for user `dd1d89a64c274cefb03a68c3233cfa51`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:04.626406 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8e0d4374-4a93-4726-b42d-cabf14852737 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:04.671341 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:04.674054 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78260aa7-5392-4144-b819-e71bdc069207 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:04.674684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78260aa7-5392-4144-b819-e71bdc069207 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:04.674962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78260aa7-5392-4144-b819-e71bdc069207 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:04.675257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78260aa7-5392-4144-b819-e71bdc069207 admin admin] PATH_INFO: `/v3/users/8de06df94c7b47719ae1f8123ddb60d3` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:04.676616 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78260aa7-5392-4144-b819-e71bdc069207 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8de06df94c7b47719ae1f8123ddb60d3)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:04.678612 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78260aa7-5392-4144-b819-e71bdc069207 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:04.684258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 903/1782] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:04 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:04.685094 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0d4374-4a93-4726-b42d-cabf14852737 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:04.686367 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8e0d4374-4a93-4726-b42d-cabf14852737 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:04.700685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0d4374-4a93-4726-b42d-cabf14852737 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:04.705400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789ff5cc-b5ca-4444-9e4e-66cd99bdeedb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:04.706033 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789ff5cc-b5ca-4444-9e4e-66cd99bdeedb admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:04.706315 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789ff5cc-b5ca-4444-9e4e-66cd99bdeedb admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:04.706586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789ff5cc-b5ca-4444-9e4e-66cd99bdeedb admin admin] PATH_INFO: `/v3/users/55e1465595804fc4a0b45c0c0ed53a2f` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:04.708071 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789ff5cc-b5ca-4444-9e4e-66cd99bdeedb admin admin] RBAC: Authorizing `identity:delete_user(user_id=55e1465595804fc4a0b45c0c0ed53a2f)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:04.709993 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789ff5cc-b5ca-4444-9e4e-66cd99bdeedb admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:04.761025 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-78260aa7-5392-4144-b819-e71bdc069207 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8de06df94c7b47719ae1f8123ddb60d3'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:04.771190 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-78260aa7-5392-4144-b819-e71bdc069207 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8de06df94c7b47719ae1f8123ddb60d3'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:04.772274 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-78260aa7-5392-4144-b819-e71bdc069207 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8de06df94c7b47719ae1f8123ddb60d3'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:04.773201 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-78260aa7-5392-4144-b819-e71bdc069207 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8de06df94c7b47719ae1f8123ddb60d3'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:04.790435 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-789ff5cc-b5ca-4444-9e4e-66cd99bdeedb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '55e1465595804fc4a0b45c0c0ed53a2f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:04.791123 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-789ff5cc-b5ca-4444-9e4e-66cd99bdeedb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '55e1465595804fc4a0b45c0c0ed53a2f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:04.794538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-78260aa7-5392-4144-b819-e71bdc069207 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8de06df94c7b47719ae1f8123ddb60d3'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:04.806921 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-789ff5cc-b5ca-4444-9e4e-66cd99bdeedb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '55e1465595804fc4a0b45c0c0ed53a2f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:04.815524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 880/1783] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:04 2022] DELETE /identity/v3/users/8de06df94c7b47719ae1f8123ddb60d3 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:04.819673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-789ff5cc-b5ca-4444-9e4e-66cd99bdeedb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '55e1465595804fc4a0b45c0c0ed53a2f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:04.831044 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-789ff5cc-b5ca-4444-9e4e-66cd99bdeedb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '55e1465595804fc4a0b45c0c0ed53a2f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:04.841219 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 904/1784] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:04 2022] DELETE /identity/v3/users/55e1465595804fc4a0b45c0c0ed53a2f => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:04.845859 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a7a8be-4f30-42ea-9d06-8268517d220c None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:04.846152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a7a8be-4f30-42ea-9d06-8268517d220c None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:04.846428 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a7a8be-4f30-42ea-9d06-8268517d220c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:04.881946 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-b5a7a8be-4f30-42ea-9d06-8268517d220c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:04.883973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-b5a7a8be-4f30-42ea-9d06-8268517d220c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:04.941537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:04.991624 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 881/1785] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:05.018452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789ff5cc-b5ca-4444-9e4e-66cd99bdeedb admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:05.020550 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-789ff5cc-b5ca-4444-9e4e-66cd99bdeedb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:05.136736 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789ff5cc-b5ca-4444-9e4e-66cd99bdeedb admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:05.141305 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2afb78-8fb3-4cfc-aebd-8616905dc2d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:05.142675 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2afb78-8fb3-4cfc-aebd-8616905dc2d2 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:05.143470 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2afb78-8fb3-4cfc-aebd-8616905dc2d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:05.144557 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2afb78-8fb3-4cfc-aebd-8616905dc2d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:05.158192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2afb78-8fb3-4cfc-aebd-8616905dc2d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:05.159801 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2afb78-8fb3-4cfc-aebd-8616905dc2d2 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:05.171451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 905/1786] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:58:05.341648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a7a8be-4f30-42ea-9d06-8268517d220c None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:05.343031 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-b5a7a8be-4f30-42ea-9d06-8268517d220c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:05.365087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a7a8be-4f30-42ea-9d06-8268517d220c None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:05.372289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13cf4ec-3b81-4a57-b976-119a47ada0cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:05.373045 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13cf4ec-3b81-4a57-b976-119a47ada0cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:05.373621 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13cf4ec-3b81-4a57-b976-119a47ada0cf admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:05.374215 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13cf4ec-3b81-4a57-b976-119a47ada0cf admin admin] PATH_INFO: `/v3/projects/9dd801ad6a7a43ef988331158da99875` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:05.376492 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13cf4ec-3b81-4a57-b976-119a47ada0cf admin admin] RBAC: Authorizing `identity:delete_project(project_id=9dd801ad6a7a43ef988331158da99875)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:05.379313 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13cf4ec-3b81-4a57-b976-119a47ada0cf admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:05.478575 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b13cf4ec-3b81-4a57-b976-119a47ada0cf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9dd801ad6a7a43ef988331158da99875', 'request_id': 'req-b13cf4ec-3b81-4a57-b976-119a47ada0cf'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:05.490515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b13cf4ec-3b81-4a57-b976-119a47ada0cf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9dd801ad6a7a43ef988331158da99875', 'request_id': 'req-b13cf4ec-3b81-4a57-b976-119a47ada0cf'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:05.501738 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b13cf4ec-3b81-4a57-b976-119a47ada0cf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9dd801ad6a7a43ef988331158da99875', 'request_id': 'req-b13cf4ec-3b81-4a57-b976-119a47ada0cf'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:05.521571 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b13cf4ec-3b81-4a57-b976-119a47ada0cf admin admin] The token cache is being invalidate because project 9dd801ad6a7a43ef988331158da99875 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:05.521915 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b13cf4ec-3b81-4a57-b976-119a47ada0cf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:05.522694 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b13cf4ec-3b81-4a57-b976-119a47ada0cf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:05.524986 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 882/1787] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:05 2022] DELETE /identity/v3/projects/9dd801ad6a7a43ef988331158da99875 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:05.530392 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387091d2-0bbd-4b56-b4ac-76120f77d385 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:05.531501 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387091d2-0bbd-4b56-b4ac-76120f77d385 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:05.531951 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387091d2-0bbd-4b56-b4ac-76120f77d385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:05.558019 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-387091d2-0bbd-4b56-b4ac-76120f77d385 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:05.559517 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-387091d2-0bbd-4b56-b4ac-76120f77d385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:05.633399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 906/1788] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:05.641195 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79da7021-8918-419c-ba85-0bf14451aff3 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:05.641874 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79da7021-8918-419c-ba85-0bf14451aff3 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:05.642453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79da7021-8918-419c-ba85-0bf14451aff3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:05.667197 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-79da7021-8918-419c-ba85-0bf14451aff3 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:05.668548 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-79da7021-8918-419c-ba85-0bf14451aff3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:05.680484 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 883/1789] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:05.685811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387091d2-0bbd-4b56-b4ac-76120f77d385 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:05.700873 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387091d2-0bbd-4b56-b4ac-76120f77d385 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:05.705898 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6707f6-7023-4788-9194-c8552468efa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:05.707289 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6707f6-7023-4788-9194-c8552468efa4 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:05.707634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6707f6-7023-4788-9194-c8552468efa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:05.708050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6707f6-7023-4788-9194-c8552468efa4 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:05.708827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6707f6-7023-4788-9194-c8552468efa4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:58:05.709247 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6707f6-7023-4788-9194-c8552468efa4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:05.710939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6707f6-7023-4788-9194-c8552468efa4 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:05.718858 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:05.730292 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 907/1790] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:58:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:58:05.731765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79da7021-8918-419c-ba85-0bf14451aff3 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:05.745055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79da7021-8918-419c-ba85-0bf14451aff3 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:05.751366 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be520c92-c766-417b-8248-e90ffd0ed0a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:05.752255 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be520c92-c766-417b-8248-e90ffd0ed0a3 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:05.752667 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be520c92-c766-417b-8248-e90ffd0ed0a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:05.753323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be520c92-c766-417b-8248-e90ffd0ed0a3 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:05.754545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be520c92-c766-417b-8248-e90ffd0ed0a3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:05.767751 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be520c92-c766-417b-8248-e90ffd0ed0a3 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:05.803928 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 884/1791] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:58:05 2022] POST /identity/v3/projects => generated 355 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:05.806883 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6707f6-7023-4788-9194-c8552468efa4 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:05.824988 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6707f6-7023-4788-9194-c8552468efa4 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:05.844154 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be520c92-c766-417b-8248-e90ffd0ed0a3 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:05.846288 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-be520c92-c766-417b-8248-e90ffd0ed0a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:05.908997 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e98ff3-38c5-4df1-86c1-5c33f83a4d56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:05.909616 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e98ff3-38c5-4df1-86c1-5c33f83a4d56 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:05.909897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e98ff3-38c5-4df1-86c1-5c33f83a4d56 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:05.910173 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e98ff3-38c5-4df1-86c1-5c33f83a4d56 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:05.910815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e98ff3-38c5-4df1-86c1-5c33f83a4d56 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:05.912882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e98ff3-38c5-4df1-86c1-5c33f83a4d56 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:05.927114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be520c92-c766-417b-8248-e90ffd0ed0a3 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:05.933302 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ab1dbc-aa2c-4712-bf15-f4175142cc09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:05.934024 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ab1dbc-aa2c-4712-bf15-f4175142cc09 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:05.934313 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ab1dbc-aa2c-4712-bf15-f4175142cc09 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:05.934765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ab1dbc-aa2c-4712-bf15-f4175142cc09 admin admin] PATH_INFO: `/v3/projects/6ba24f5a0f4b40c498bb15e73bd92515` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:05.936223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ab1dbc-aa2c-4712-bf15-f4175142cc09 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ba24f5a0f4b40c498bb15e73bd92515)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:05.938154 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ab1dbc-aa2c-4712-bf15-f4175142cc09 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:05.943607 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 908/1792] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:58:05 2022] POST /identity/v3/users => generated 278 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:05.947339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e98ff3-38c5-4df1-86c1-5c33f83a4d56 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:05.960476 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e98ff3-38c5-4df1-86c1-5c33f83a4d56 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:05.967663 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8105ef-de0e-430e-a19d-8268c69ddfa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:05.968279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8105ef-de0e-430e-a19d-8268c69ddfa4 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:05.968563 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8105ef-de0e-430e-a19d-8268c69ddfa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:05.968870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8105ef-de0e-430e-a19d-8268c69ddfa4 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:05.969517 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8105ef-de0e-430e-a19d-8268c69ddfa4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:05.971137 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8105ef-de0e-430e-a19d-8268c69ddfa4 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:05.984906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 909/1793] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:05 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:05.985446 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:05.988895 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8105ef-de0e-430e-a19d-8268c69ddfa4 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:06.000690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8105ef-de0e-430e-a19d-8268c69ddfa4 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:06.005142 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-71ab1dbc-aa2c-4712-bf15-f4175142cc09 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ba24f5a0f4b40c498bb15e73bd92515', 'request_id': 'req-71ab1dbc-aa2c-4712-bf15-f4175142cc09'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:06.015498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-71ab1dbc-aa2c-4712-bf15-f4175142cc09 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ba24f5a0f4b40c498bb15e73bd92515', 'request_id': 'req-71ab1dbc-aa2c-4712-bf15-f4175142cc09'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:06.022329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-71ab1dbc-aa2c-4712-bf15-f4175142cc09 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ba24f5a0f4b40c498bb15e73bd92515', 'request_id': 'req-71ab1dbc-aa2c-4712-bf15-f4175142cc09'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:06.036638 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-71ab1dbc-aa2c-4712-bf15-f4175142cc09 admin admin] The token cache is being invalidate because project 6ba24f5a0f4b40c498bb15e73bd92515 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:06.037034 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-71ab1dbc-aa2c-4712-bf15-f4175142cc09 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:06.037787 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-71ab1dbc-aa2c-4712-bf15-f4175142cc09 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:06.040299 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 885/1794] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:05 2022] DELETE /identity/v3/projects/6ba24f5a0f4b40c498bb15e73bd92515 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:58:06.043904 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ab1dbc-aa2c-4712-bf15-f4175142cc09 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:06.045352 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-71ab1dbc-aa2c-4712-bf15-f4175142cc09 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:06.073140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf15e49-0e2b-45fa-b034-4c72c1f96939 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:06.073889 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf15e49-0e2b-45fa-b034-4c72c1f96939 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:06.074211 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf15e49-0e2b-45fa-b034-4c72c1f96939 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:06.074554 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf15e49-0e2b-45fa-b034-4c72c1f96939 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:06.075260 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf15e49-0e2b-45fa-b034-4c72c1f96939 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:06.076951 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf15e49-0e2b-45fa-b034-4c72c1f96939 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:06.088729 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 910/1795] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:05 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 14:58:06.093125 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf15e49-0e2b-45fa-b034-4c72c1f96939 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:06.094742 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-caf15e49-0e2b-45fa-b034-4c72c1f96939 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:06.108134 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf15e49-0e2b-45fa-b034-4c72c1f96939 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:06.113391 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2623f3bf-f855-4864-8a03-74d3075189e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:06.114052 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2623f3bf-f855-4864-8a03-74d3075189e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:06.114333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2623f3bf-f855-4864-8a03-74d3075189e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:06.114607 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2623f3bf-f855-4864-8a03-74d3075189e4 admin admin] PATH_INFO: `/v3/projects/0a03491a2ffc4855a9a18f173f81ca4b/users/eae1a3578c7343d39147e210369a9b46/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:06.125779 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ab1dbc-aa2c-4712-bf15-f4175142cc09 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:06.130643 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cbe29c-42bc-451c-b3fa-0650ece3009e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:06.130820 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2623f3bf-f855-4864-8a03-74d3075189e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a03491a2ffc4855a9a18f173f81ca4b, user_id=eae1a3578c7343d39147e210369a9b46, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:06.131270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cbe29c-42bc-451c-b3fa-0650ece3009e admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:06.131570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cbe29c-42bc-451c-b3fa-0650ece3009e admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:06.131851 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cbe29c-42bc-451c-b3fa-0650ece3009e admin admin] PATH_INFO: `/v3/users/019409f45b174140bd7605f7e9ad088d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:06.133043 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cbe29c-42bc-451c-b3fa-0650ece3009e admin admin] RBAC: Authorizing `identity:delete_user(user_id=019409f45b174140bd7605f7e9ad088d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:06.133742 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2623f3bf-f855-4864-8a03-74d3075189e4 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:06.135123 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cbe29c-42bc-451c-b3fa-0650ece3009e admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:06.152510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 911/1796] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:06 2022] PUT /identity/v3/projects/0a03491a2ffc4855a9a18f173f81ca4b/users/eae1a3578c7343d39147e210369a9b46/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:06.162270 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77a9cff-190a-4707-aadb-dd757121e054 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:06.162735 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77a9cff-190a-4707-aadb-dd757121e054 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:06.163182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77a9cff-190a-4707-aadb-dd757121e054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:06.198960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-f77a9cff-190a-4707-aadb-dd757121e054 None None] MFA Rules not processed for user `eae1a3578c7343d39147e210369a9b46`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:06.200761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-f77a9cff-190a-4707-aadb-dd757121e054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:06.204987 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-47cbe29c-42bc-451c-b3fa-0650ece3009e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '019409f45b174140bd7605f7e9ad088d'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:06.217137 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-47cbe29c-42bc-451c-b3fa-0650ece3009e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '019409f45b174140bd7605f7e9ad088d'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:06.219657 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-47cbe29c-42bc-451c-b3fa-0650ece3009e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '019409f45b174140bd7605f7e9ad088d'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:06.220363 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-47cbe29c-42bc-451c-b3fa-0650ece3009e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '019409f45b174140bd7605f7e9ad088d'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:06.245861 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-47cbe29c-42bc-451c-b3fa-0650ece3009e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '019409f45b174140bd7605f7e9ad088d'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:06.251518 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:06.261446 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 886/1797] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:06 2022] DELETE /identity/v3/users/019409f45b174140bd7605f7e9ad088d => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:58:06.317469 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 912/1798] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:06 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:06.318978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cbe29c-42bc-451c-b3fa-0650ece3009e admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:06.320986 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-47cbe29c-42bc-451c-b3fa-0650ece3009e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:06.378855 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:06.408404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cbe29c-42bc-451c-b3fa-0650ece3009e admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:06.413333 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401a4a97-7ffa-41eb-88ac-5a8968594312 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:06.414033 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401a4a97-7ffa-41eb-88ac-5a8968594312 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:06.414724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401a4a97-7ffa-41eb-88ac-5a8968594312 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:06.414871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401a4a97-7ffa-41eb-88ac-5a8968594312 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:06.415810 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401a4a97-7ffa-41eb-88ac-5a8968594312 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:06.421370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401a4a97-7ffa-41eb-88ac-5a8968594312 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:06.454758 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 887/1799] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:58:06 2022] POST /identity/v3/projects => generated 357 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:58:06.458898 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77a9cff-190a-4707-aadb-dd757121e054 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:06.548429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77a9cff-190a-4707-aadb-dd757121e054 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:06.585893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401a4a97-7ffa-41eb-88ac-5a8968594312 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:06.588289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-401a4a97-7ffa-41eb-88ac-5a8968594312 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:06.627866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3464e106-7180-44f9-9bcc-89eab5fc29c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:06.628626 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3464e106-7180-44f9-9bcc-89eab5fc29c0 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:06.628917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3464e106-7180-44f9-9bcc-89eab5fc29c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:06.629396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3464e106-7180-44f9-9bcc-89eab5fc29c0 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:06.630257 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3464e106-7180-44f9-9bcc-89eab5fc29c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:06.640156 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3464e106-7180-44f9-9bcc-89eab5fc29c0 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:06.668625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 913/1800] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:58:06 2022] POST /identity/v3/users => generated 285 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:58:06.672629 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3464e106-7180-44f9-9bcc-89eab5fc29c0 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:06.685583 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3464e106-7180-44f9-9bcc-89eab5fc29c0 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:06.686528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401a4a97-7ffa-41eb-88ac-5a8968594312 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:06.694171 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf83371-8cc7-4934-af01-1a5c9f93a184 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:06.694915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf83371-8cc7-4934-af01-1a5c9f93a184 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:06.695799 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf83371-8cc7-4934-af01-1a5c9f93a184 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:06.696180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf83371-8cc7-4934-af01-1a5c9f93a184 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:06.697513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf83371-8cc7-4934-af01-1a5c9f93a184 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:06.699348 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf83371-8cc7-4934-af01-1a5c9f93a184 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:06.703537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb74f8dc-921b-4bd5-ae8e-7073039645b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:06.704663 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb74f8dc-921b-4bd5-ae8e-7073039645b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:06.705179 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb74f8dc-921b-4bd5-ae8e-7073039645b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:06.705667 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb74f8dc-921b-4bd5-ae8e-7073039645b3 admin admin] PATH_INFO: `/v3/projects/9b6e9bd8b6c24bd2bb50f8df8ac2e38b` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:06.707889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb74f8dc-921b-4bd5-ae8e-7073039645b3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9b6e9bd8b6c24bd2bb50f8df8ac2e38b)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:06.711684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb74f8dc-921b-4bd5-ae8e-7073039645b3 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:06.717779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 914/1801] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:06 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:06.719909 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf83371-8cc7-4934-af01-1a5c9f93a184 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:06.734279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf83371-8cc7-4934-af01-1a5c9f93a184 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:06.753247 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2a009d-824e-4f00-a5ab-390cf0212b41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:06.753977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2a009d-824e-4f00-a5ab-390cf0212b41 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:06.756311 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2a009d-824e-4f00-a5ab-390cf0212b41 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:06.756714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2a009d-824e-4f00-a5ab-390cf0212b41 admin admin] PATH_INFO: `/v3/projects/69e2d5d6ea91469ea8f4562f699f3e42/users/ea82841325a4490fada61317690a1456/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:06.778915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2a009d-824e-4f00-a5ab-390cf0212b41 admin admin] RBAC: Authorizing `identity:create_grant(project_id=69e2d5d6ea91469ea8f4562f699f3e42, user_id=ea82841325a4490fada61317690a1456, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:06.782183 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2a009d-824e-4f00-a5ab-390cf0212b41 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:06.797241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-cb74f8dc-921b-4bd5-ae8e-7073039645b3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9b6e9bd8b6c24bd2bb50f8df8ac2e38b', 'request_id': 'req-cb74f8dc-921b-4bd5-ae8e-7073039645b3'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:06.805784 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 915/1802] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:06 2022] PUT /identity/v3/projects/69e2d5d6ea91469ea8f4562f699f3e42/users/ea82841325a4490fada61317690a1456/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:58:06.809783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2a009d-824e-4f00-a5ab-390cf0212b41 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:06.818511 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-cb74f8dc-921b-4bd5-ae8e-7073039645b3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9b6e9bd8b6c24bd2bb50f8df8ac2e38b', 'request_id': 'req-cb74f8dc-921b-4bd5-ae8e-7073039645b3'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:06.833719 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-cb74f8dc-921b-4bd5-ae8e-7073039645b3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9b6e9bd8b6c24bd2bb50f8df8ac2e38b', 'request_id': 'req-cb74f8dc-921b-4bd5-ae8e-7073039645b3'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:06.853712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-cb74f8dc-921b-4bd5-ae8e-7073039645b3 admin admin] The token cache is being invalidate because project 9b6e9bd8b6c24bd2bb50f8df8ac2e38b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:06.854011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-cb74f8dc-921b-4bd5-ae8e-7073039645b3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:06.855385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-cb74f8dc-921b-4bd5-ae8e-7073039645b3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:06.860274 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 888/1803] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:06 2022] DELETE /identity/v3/projects/9b6e9bd8b6c24bd2bb50f8df8ac2e38b => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:58:06.864535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb74f8dc-921b-4bd5-ae8e-7073039645b3 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:06.866475 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-cb74f8dc-921b-4bd5-ae8e-7073039645b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:06.870531 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:06.898590 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2a009d-824e-4f00-a5ab-390cf0212b41 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:06.905749 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c279a1f0-4292-45f4-89cf-896bc5db3a4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:06.906407 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c279a1f0-4292-45f4-89cf-896bc5db3a4e admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:06.906686 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c279a1f0-4292-45f4-89cf-896bc5db3a4e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:06.906957 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c279a1f0-4292-45f4-89cf-896bc5db3a4e admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:06.907687 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c279a1f0-4292-45f4-89cf-896bc5db3a4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:06.909431 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c279a1f0-4292-45f4-89cf-896bc5db3a4e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:06.925354 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 916/1804] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:06 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:06.926228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c279a1f0-4292-45f4-89cf-896bc5db3a4e admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:06.928018 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-c279a1f0-4292-45f4-89cf-896bc5db3a4e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:06.941678 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c279a1f0-4292-45f4-89cf-896bc5db3a4e admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:06.942176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:06.948079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c93b7e-ea58-4d4e-95a2-e8c9d9873eb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:06.949053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c93b7e-ea58-4d4e-95a2-e8c9d9873eb4 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:06.949053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c93b7e-ea58-4d4e-95a2-e8c9d9873eb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:06.949315 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c93b7e-ea58-4d4e-95a2-e8c9d9873eb4 admin admin] PATH_INFO: `/v3/projects/69e2d5d6ea91469ea8f4562f699f3e42/users/ea82841325a4490fada61317690a1456/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:06.951389 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c93b7e-ea58-4d4e-95a2-e8c9d9873eb4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=69e2d5d6ea91469ea8f4562f699f3e42, user_id=ea82841325a4490fada61317690a1456, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:06.954124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c93b7e-ea58-4d4e-95a2-e8c9d9873eb4 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:06.957630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb74f8dc-921b-4bd5-ae8e-7073039645b3 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:06.966358 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f2aec6-a74c-4518-b173-94e3b502452e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:06.967040 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f2aec6-a74c-4518-b173-94e3b502452e admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:06.967323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f2aec6-a74c-4518-b173-94e3b502452e admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:06.967619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f2aec6-a74c-4518-b173-94e3b502452e admin admin] PATH_INFO: `/v3/users/2a460b856413424687069a18b5c0e347` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:06.967839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 917/1805] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:06 2022] PUT /identity/v3/projects/69e2d5d6ea91469ea8f4562f699f3e42/users/ea82841325a4490fada61317690a1456/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:06.968900 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f2aec6-a74c-4518-b173-94e3b502452e admin admin] RBAC: Authorizing `identity:delete_user(user_id=2a460b856413424687069a18b5c0e347)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:06.970884 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f2aec6-a74c-4518-b173-94e3b502452e admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:06.980074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d161a3da-a6c8-4c3e-90ae-de12115dc839 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:06.980424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d161a3da-a6c8-4c3e-90ae-de12115dc839 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:06.980857 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d161a3da-a6c8-4c3e-90ae-de12115dc839 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:07.019972 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-d161a3da-a6c8-4c3e-90ae-de12115dc839 None None] MFA Rules not processed for user `ea82841325a4490fada61317690a1456`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:07.021508 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d161a3da-a6c8-4c3e-90ae-de12115dc839 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:07.027989 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-04f2aec6-a74c-4518-b173-94e3b502452e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2a460b856413424687069a18b5c0e347'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:07.036054 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-04f2aec6-a74c-4518-b173-94e3b502452e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2a460b856413424687069a18b5c0e347'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:07.036782 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-04f2aec6-a74c-4518-b173-94e3b502452e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2a460b856413424687069a18b5c0e347'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:07.037412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-04f2aec6-a74c-4518-b173-94e3b502452e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2a460b856413424687069a18b5c0e347'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:07.056175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-04f2aec6-a74c-4518-b173-94e3b502452e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2a460b856413424687069a18b5c0e347'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:07.074096 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 889/1806] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:06 2022] DELETE /identity/v3/users/2a460b856413424687069a18b5c0e347 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:07.113876 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 918/1807] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:06 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:07.126053 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f2aec6-a74c-4518-b173-94e3b502452e admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:07.127521 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-04f2aec6-a74c-4518-b173-94e3b502452e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:07.214698 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f2aec6-a74c-4518-b173-94e3b502452e admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:07.218923 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4d41e4-4aee-422b-8d22-bd79ebda943b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:07.220157 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4d41e4-4aee-422b-8d22-bd79ebda943b service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:07.220157 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4d41e4-4aee-422b-8d22-bd79ebda943b service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:07.220157 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4d41e4-4aee-422b-8d22-bd79ebda943b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:07.300945 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4d41e4-4aee-422b-8d22-bd79ebda943b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:07.302370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4d41e4-4aee-422b-8d22-bd79ebda943b service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:07.313647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 890/1808] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:07 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:07.423655 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c37fa4-b2c1-4b11-b89e-1586f61048db None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:07.424280 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c37fa4-b2c1-4b11-b89e-1586f61048db None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:07.424280 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c37fa4-b2c1-4b11-b89e-1586f61048db None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:07.425804 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 919/1809] 10.222.0.36 () {54 vars in 921 bytes} [Mon Oct 31 14:58:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:58:07.438638 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0742ee0-5da6-43e0-89f1-f3ed7e2ee949 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:07.439192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0742ee0-5da6-43e0-89f1-f3ed7e2ee949 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:07.439588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0742ee0-5da6-43e0-89f1-f3ed7e2ee949 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:07.445952 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c37fa4-b2c1-4b11-b89e-1586f61048db None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:07.466054 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c37fa4-b2c1-4b11-b89e-1586f61048db None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:07.474546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b303fcc7-8f66-4246-90a5-c482ed40d45c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:07.475245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b303fcc7-8f66-4246-90a5-c482ed40d45c service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:07.476128 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b303fcc7-8f66-4246-90a5-c482ed40d45c service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:07.477428 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b303fcc7-8f66-4246-90a5-c482ed40d45c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:07.478285 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-a0742ee0-5da6-43e0-89f1-f3ed7e2ee949 None None] MFA Rules not processed for user `70ea2aaf529b46dabb70b71f4dcc3860`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:07.479947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a0742ee0-5da6-43e0-89f1-f3ed7e2ee949 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:07.480620 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b303fcc7-8f66-4246-90a5-c482ed40d45c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:07.504620 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:07.587119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 891/1810] 10.222.0.36 () {58 vars in 1018 bytes} [Mon Oct 31 14:58:07 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:07.589050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b303fcc7-8f66-4246-90a5-c482ed40d45c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:07.590521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b303fcc7-8f66-4246-90a5-c482ed40d45c service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:07.603186 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 920/1811] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:07 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:58:07.611817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0742ee0-5da6-43e0-89f1-f3ed7e2ee949 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:07.614803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a0742ee0-5da6-43e0-89f1-f3ed7e2ee949 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:07.636810 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0742ee0-5da6-43e0-89f1-f3ed7e2ee949 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:07.644486 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269d294d-667a-4888-b27f-2f9efccdff00 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70ea2aaf529b46dabb70b71f4dcc3860', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:07.645108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269d294d-667a-4888-b27f-2f9efccdff00 service placement] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:07.646238 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269d294d-667a-4888-b27f-2f9efccdff00 service placement] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:07.646534 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269d294d-667a-4888-b27f-2f9efccdff00 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:07.659600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269d294d-667a-4888-b27f-2f9efccdff00 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:07.661694 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269d294d-667a-4888-b27f-2f9efccdff00 service placement] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:07.674970 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 892/1812] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:07 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:07.846344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b303fcc7-8f66-4246-90a5-c482ed40d45c service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:07.848624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b303fcc7-8f66-4246-90a5-c482ed40d45c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:07.925043 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:07.950745 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b303fcc7-8f66-4246-90a5-c482ed40d45c service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:07.957497 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb32eca9-45f4-413b-91f4-bd6fb08f7faf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:07.958158 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb32eca9-45f4-413b-91f4-bd6fb08f7faf admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:07.959032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb32eca9-45f4-413b-91f4-bd6fb08f7faf admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:07.959510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb32eca9-45f4-413b-91f4-bd6fb08f7faf admin admin] PATH_INFO: `/v3/projects/8146932fa3e3458f914b0b531dcdaa98` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:07.966405 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb32eca9-45f4-413b-91f4-bd6fb08f7faf admin admin] RBAC: Authorizing `identity:delete_project(project_id=8146932fa3e3458f914b0b531dcdaa98)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:07.966405 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb32eca9-45f4-413b-91f4-bd6fb08f7faf admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:08.026167 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-bb32eca9-45f4-413b-91f4-bd6fb08f7faf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8146932fa3e3458f914b0b531dcdaa98', 'request_id': 'req-bb32eca9-45f4-413b-91f4-bd6fb08f7faf'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.033260 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-bb32eca9-45f4-413b-91f4-bd6fb08f7faf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8146932fa3e3458f914b0b531dcdaa98', 'request_id': 'req-bb32eca9-45f4-413b-91f4-bd6fb08f7faf'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.040474 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-bb32eca9-45f4-413b-91f4-bd6fb08f7faf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8146932fa3e3458f914b0b531dcdaa98', 'request_id': 'req-bb32eca9-45f4-413b-91f4-bd6fb08f7faf'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.055312 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-bb32eca9-45f4-413b-91f4-bd6fb08f7faf admin admin] The token cache is being invalidate because project 8146932fa3e3458f914b0b531dcdaa98 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:08.055312 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-bb32eca9-45f4-413b-91f4-bd6fb08f7faf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.056030 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-bb32eca9-45f4-413b-91f4-bd6fb08f7faf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.058112 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 921/1813] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:07 2022] DELETE /identity/v3/projects/8146932fa3e3458f914b0b531dcdaa98 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:08.062967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58e673a-ec74-463e-b4c6-ccf9c6fe258c None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:08.063606 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58e673a-ec74-463e-b4c6-ccf9c6fe258c None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:08.065547 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58e673a-ec74-463e-b4c6-ccf9c6fe258c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:08.097181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-d58e673a-ec74-463e-b4c6-ccf9c6fe258c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:08.098424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d58e673a-ec74-463e-b4c6-ccf9c6fe258c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:08.174340 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 893/1814] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:08.176168 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98b397-e53b-4db4-8573-2ca53a8d4df7 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:08.176520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98b397-e53b-4db4-8573-2ca53a8d4df7 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:08.176718 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98b397-e53b-4db4-8573-2ca53a8d4df7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:08.205746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58e673a-ec74-463e-b4c6-ccf9c6fe258c None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:08.207594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d58e673a-ec74-463e-b4c6-ccf9c6fe258c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:08.213410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-2f98b397-e53b-4db4-8573-2ca53a8d4df7 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:08.215935 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2f98b397-e53b-4db4-8573-2ca53a8d4df7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:08.222420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58e673a-ec74-463e-b4c6-ccf9c6fe258c None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:08.227289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd348e0-67a4-4a9e-9367-1ca816c135f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:08.227943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd348e0-67a4-4a9e-9367-1ca816c135f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:08.228227 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd348e0-67a4-4a9e-9367-1ca816c135f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:08.228505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd348e0-67a4-4a9e-9367-1ca816c135f8 admin admin] PATH_INFO: `/v3/users/2b003dae54c74a9784b1d76e8388a2c0` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:08.230362 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd348e0-67a4-4a9e-9367-1ca816c135f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2b003dae54c74a9784b1d76e8388a2c0)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:08.230911 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 922/1815] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:08.232767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd348e0-67a4-4a9e-9367-1ca816c135f8 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:08.235213 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f98b397-e53b-4db4-8573-2ca53a8d4df7 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:08.247212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f98b397-e53b-4db4-8573-2ca53a8d4df7 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:08.253722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164aee65-bd59-4303-a3e8-ed67d8645f51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:08.254845 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164aee65-bd59-4303-a3e8-ed67d8645f51 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:08.254845 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164aee65-bd59-4303-a3e8-ed67d8645f51 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:08.254845 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164aee65-bd59-4303-a3e8-ed67d8645f51 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:08.255495 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164aee65-bd59-4303-a3e8-ed67d8645f51 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:58:08.255832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164aee65-bd59-4303-a3e8-ed67d8645f51 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:08.257437 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164aee65-bd59-4303-a3e8-ed67d8645f51 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:08.273405 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 923/1816] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:58:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:58:08.278149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164aee65-bd59-4303-a3e8-ed67d8645f51 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:08.290175 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164aee65-bd59-4303-a3e8-ed67d8645f51 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:08.295775 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2bd348e0-67a4-4a9e-9367-1ca816c135f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2b003dae54c74a9784b1d76e8388a2c0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.297270 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9bb17e-d09a-4f98-9091-e3532aa7c61b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:08.298531 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9bb17e-d09a-4f98-9091-e3532aa7c61b admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:08.299024 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9bb17e-d09a-4f98-9091-e3532aa7c61b admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:08.299147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9bb17e-d09a-4f98-9091-e3532aa7c61b admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:08.299947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9bb17e-d09a-4f98-9091-e3532aa7c61b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:08.302781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9bb17e-d09a-4f98-9091-e3532aa7c61b admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:08.307032 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2bd348e0-67a4-4a9e-9367-1ca816c135f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2b003dae54c74a9784b1d76e8388a2c0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.308103 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2bd348e0-67a4-4a9e-9367-1ca816c135f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2b003dae54c74a9784b1d76e8388a2c0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.309790 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2bd348e0-67a4-4a9e-9367-1ca816c135f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2b003dae54c74a9784b1d76e8388a2c0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.327161 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2bd348e0-67a4-4a9e-9367-1ca816c135f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2b003dae54c74a9784b1d76e8388a2c0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.340228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 924/1817] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:58:08 2022] POST /identity/v3/projects => generated 373 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:08.341640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9bb17e-d09a-4f98-9091-e3532aa7c61b admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:08.343495 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ea9bb17e-d09a-4f98-9091-e3532aa7c61b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:08.343627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 894/1818] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:08 2022] DELETE /identity/v3/users/2b003dae54c74a9784b1d76e8388a2c0 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:58:08.411657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9bb17e-d09a-4f98-9091-e3532aa7c61b admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:08.455145 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:08.477710 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c19b414-2fc4-42ac-8fdc-b33a4e98365c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:08.478451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c19b414-2fc4-42ac-8fdc-b33a4e98365c admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:08.478607 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c19b414-2fc4-42ac-8fdc-b33a4e98365c admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:08.478879 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c19b414-2fc4-42ac-8fdc-b33a4e98365c admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:08.479837 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c19b414-2fc4-42ac-8fdc-b33a4e98365c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:08.481779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c19b414-2fc4-42ac-8fdc-b33a4e98365c admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:08.510268 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 925/1819] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:58:08 2022] POST /identity/v3/users => generated 287 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:58:08.511832 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd348e0-67a4-4a9e-9367-1ca816c135f8 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:08.524666 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd348e0-67a4-4a9e-9367-1ca816c135f8 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:08.531280 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e6f45d-b583-4d5f-881f-028f15008551 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:08.532574 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e6f45d-b583-4d5f-881f-028f15008551 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:08.532907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e6f45d-b583-4d5f-881f-028f15008551 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:08.533181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e6f45d-b583-4d5f-881f-028f15008551 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:08.534131 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e6f45d-b583-4d5f-881f-028f15008551 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:08.536160 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e6f45d-b583-4d5f-881f-028f15008551 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:08.555664 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 895/1820] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:08 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:08.558440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c19b414-2fc4-42ac-8fdc-b33a4e98365c admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:08.569778 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c19b414-2fc4-42ac-8fdc-b33a4e98365c admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:08.576924 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165846d3-2c1e-4d6c-b553-e92a7d61b273 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:08.578327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e6f45d-b583-4d5f-881f-028f15008551 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:08.580147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165846d3-2c1e-4d6c-b553-e92a7d61b273 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:08.580366 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-70e6f45d-b583-4d5f-881f-028f15008551 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:08.580817 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165846d3-2c1e-4d6c-b553-e92a7d61b273 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:08.581286 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165846d3-2c1e-4d6c-b553-e92a7d61b273 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:08.582445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165846d3-2c1e-4d6c-b553-e92a7d61b273 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:08.584843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165846d3-2c1e-4d6c-b553-e92a7d61b273 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:08.599019 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 926/1821] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:08 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:08.604188 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165846d3-2c1e-4d6c-b553-e92a7d61b273 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:08.607514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e6f45d-b583-4d5f-881f-028f15008551 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:08.614988 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a597f10e-04ff-4e06-b875-c61ac1976821 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:08.615827 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a597f10e-04ff-4e06-b875-c61ac1976821 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:08.616093 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a597f10e-04ff-4e06-b875-c61ac1976821 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:08.616448 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a597f10e-04ff-4e06-b875-c61ac1976821 admin admin] PATH_INFO: `/v3/projects/b862ff186521426dbf0da8a38198d645` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:08.618138 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165846d3-2c1e-4d6c-b553-e92a7d61b273 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:08.618258 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a597f10e-04ff-4e06-b875-c61ac1976821 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b862ff186521426dbf0da8a38198d645)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:08.621179 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a597f10e-04ff-4e06-b875-c61ac1976821 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:08.623363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3f1961-958b-4e5c-afcd-e23ff06a9ff1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:08.623793 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:08.624446 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3f1961-958b-4e5c-afcd-e23ff06a9ff1 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:08.625041 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3f1961-958b-4e5c-afcd-e23ff06a9ff1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:08.625462 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3f1961-958b-4e5c-afcd-e23ff06a9ff1 admin admin] PATH_INFO: `/v3/projects/ba40f7e353be44918e39d72cabf4198c/users/79f38f4b488f417f80f46ec4b156e65a/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:08.642672 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3f1961-958b-4e5c-afcd-e23ff06a9ff1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba40f7e353be44918e39d72cabf4198c, user_id=79f38f4b488f417f80f46ec4b156e65a, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:08.645870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3f1961-958b-4e5c-afcd-e23ff06a9ff1 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:08.664696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 927/1822] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:08 2022] PUT /identity/v3/projects/ba40f7e353be44918e39d72cabf4198c/users/79f38f4b488f417f80f46ec4b156e65a/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:08.675254 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044fe37f-f6d1-4b29-a51e-8a422e7e3ddd None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:08.675821 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044fe37f-f6d1-4b29-a51e-8a422e7e3ddd None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:08.676672 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044fe37f-f6d1-4b29-a51e-8a422e7e3ddd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:08.685893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a597f10e-04ff-4e06-b875-c61ac1976821 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b862ff186521426dbf0da8a38198d645', 'request_id': 'req-a597f10e-04ff-4e06-b875-c61ac1976821'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.694322 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a597f10e-04ff-4e06-b875-c61ac1976821 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b862ff186521426dbf0da8a38198d645', 'request_id': 'req-a597f10e-04ff-4e06-b875-c61ac1976821'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.701381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a597f10e-04ff-4e06-b875-c61ac1976821 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b862ff186521426dbf0da8a38198d645', 'request_id': 'req-a597f10e-04ff-4e06-b875-c61ac1976821'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.713125 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-044fe37f-f6d1-4b29-a51e-8a422e7e3ddd None None] MFA Rules not processed for user `79f38f4b488f417f80f46ec4b156e65a`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:08.714365 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-044fe37f-f6d1-4b29-a51e-8a422e7e3ddd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:08.732551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a597f10e-04ff-4e06-b875-c61ac1976821 admin admin] The token cache is being invalidate because project b862ff186521426dbf0da8a38198d645 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:08.733020 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a597f10e-04ff-4e06-b875-c61ac1976821 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.734229 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a597f10e-04ff-4e06-b875-c61ac1976821 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.738603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 896/1823] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:08 2022] DELETE /identity/v3/projects/b862ff186521426dbf0da8a38198d645 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:08.739961 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a597f10e-04ff-4e06-b875-c61ac1976821 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:08.741604 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a597f10e-04ff-4e06-b875-c61ac1976821 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:08.792307 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 928/1824] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:08 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:08.803687 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044fe37f-f6d1-4b29-a51e-8a422e7e3ddd None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:08.805519 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-044fe37f-f6d1-4b29-a51e-8a422e7e3ddd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:08.827786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a597f10e-04ff-4e06-b875-c61ac1976821 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:08.834254 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a7bc0b-8ae1-4f64-90fe-2baf504cbad6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:08.834951 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a7bc0b-8ae1-4f64-90fe-2baf504cbad6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:08.835232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a7bc0b-8ae1-4f64-90fe-2baf504cbad6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:08.835519 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a7bc0b-8ae1-4f64-90fe-2baf504cbad6 admin admin] PATH_INFO: `/v3/users/db1ccc457fe944159ef3db1e12c2c9ae` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:08.837242 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a7bc0b-8ae1-4f64-90fe-2baf504cbad6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=db1ccc457fe944159ef3db1e12c2c9ae)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:08.839298 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a7bc0b-8ae1-4f64-90fe-2baf504cbad6 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:08.888058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044fe37f-f6d1-4b29-a51e-8a422e7e3ddd None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:08.897122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65e5afa-f754-457e-b19c-87707044111b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:08.897484 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65e5afa-f754-457e-b19c-87707044111b service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:08.897975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65e5afa-f754-457e-b19c-87707044111b service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:08.898290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65e5afa-f754-457e-b19c-87707044111b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:08.908515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b9a7bc0b-8ae1-4f64-90fe-2baf504cbad6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'db1ccc457fe944159ef3db1e12c2c9ae'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.910591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65e5afa-f754-457e-b19c-87707044111b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:08.912307 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65e5afa-f754-457e-b19c-87707044111b service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:08.918542 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b9a7bc0b-8ae1-4f64-90fe-2baf504cbad6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'db1ccc457fe944159ef3db1e12c2c9ae'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.919283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b9a7bc0b-8ae1-4f64-90fe-2baf504cbad6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'db1ccc457fe944159ef3db1e12c2c9ae'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.920597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b9a7bc0b-8ae1-4f64-90fe-2baf504cbad6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'db1ccc457fe944159ef3db1e12c2c9ae'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.925514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 929/1825] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:08 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:58:08.942770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b9a7bc0b-8ae1-4f64-90fe-2baf504cbad6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'db1ccc457fe944159ef3db1e12c2c9ae'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:08.955416 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65e5afa-f754-457e-b19c-87707044111b service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:08.957452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e65e5afa-f754-457e-b19c-87707044111b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:08.971600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 897/1826] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:08 2022] DELETE /identity/v3/users/db1ccc457fe944159ef3db1e12c2c9ae => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:08.976755 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65e5afa-f754-457e-b19c-87707044111b service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:09.019372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:09.078223 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c431f7-2c29-41ad-9fb4-76e1e90bf70c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:09.078781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c431f7-2c29-41ad-9fb4-76e1e90bf70c admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:09.079370 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c431f7-2c29-41ad-9fb4-76e1e90bf70c admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:09.079370 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c431f7-2c29-41ad-9fb4-76e1e90bf70c admin admin] PATH_INFO: `/v3/users/79f38f4b488f417f80f46ec4b156e65a` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:09.080506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c431f7-2c29-41ad-9fb4-76e1e90bf70c admin admin] RBAC: Authorizing `identity:delete_user(user_id=79f38f4b488f417f80f46ec4b156e65a)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:09.083364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c431f7-2c29-41ad-9fb4-76e1e90bf70c admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:09.149569 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e9c431f7-2c29-41ad-9fb4-76e1e90bf70c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '79f38f4b488f417f80f46ec4b156e65a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:09.150260 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e9c431f7-2c29-41ad-9fb4-76e1e90bf70c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '79f38f4b488f417f80f46ec4b156e65a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:09.164636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e9c431f7-2c29-41ad-9fb4-76e1e90bf70c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '79f38f4b488f417f80f46ec4b156e65a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:09.172169 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e9c431f7-2c29-41ad-9fb4-76e1e90bf70c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '79f38f4b488f417f80f46ec4b156e65a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:09.183641 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e9c431f7-2c29-41ad-9fb4-76e1e90bf70c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '79f38f4b488f417f80f46ec4b156e65a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:09.191871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 930/1827] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:08 2022] DELETE /identity/v3/users/79f38f4b488f417f80f46ec4b156e65a => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:09.198103 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606e6e51-70f3-4d50-8299-c684e71bdcfc None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:09.199392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606e6e51-70f3-4d50-8299-c684e71bdcfc None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:09.199717 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606e6e51-70f3-4d50-8299-c684e71bdcfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:09.220472 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-606e6e51-70f3-4d50-8299-c684e71bdcfc None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:09.221701 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-606e6e51-70f3-4d50-8299-c684e71bdcfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:09.226000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:09.307645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c431f7-2c29-41ad-9fb4-76e1e90bf70c admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:09.313936 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e9c431f7-2c29-41ad-9fb4-76e1e90bf70c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:09.326180 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 898/1828] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:09.343269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c431f7-2c29-41ad-9fb4-76e1e90bf70c admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:09.345578 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606e6e51-70f3-4d50-8299-c684e71bdcfc None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:09.349091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-606e6e51-70f3-4d50-8299-c684e71bdcfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:09.353293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c519a861-dee4-475b-ac81-226e3903d765 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:09.353858 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c519a861-dee4-475b-ac81-226e3903d765 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:09.354505 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c519a861-dee4-475b-ac81-226e3903d765 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:09.354785 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c519a861-dee4-475b-ac81-226e3903d765 admin admin] PATH_INFO: `/v3/projects/53440d7b5288482da82c3c3f76d8e759` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:09.356003 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c519a861-dee4-475b-ac81-226e3903d765 admin admin] RBAC: Authorizing `identity:delete_project(project_id=53440d7b5288482da82c3c3f76d8e759)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:09.358333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c519a861-dee4-475b-ac81-226e3903d765 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:09.417242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c519a861-dee4-475b-ac81-226e3903d765 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '53440d7b5288482da82c3c3f76d8e759', 'request_id': 'req-c519a861-dee4-475b-ac81-226e3903d765'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:09.420594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606e6e51-70f3-4d50-8299-c684e71bdcfc None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:09.423412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c519a861-dee4-475b-ac81-226e3903d765 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '53440d7b5288482da82c3c3f76d8e759', 'request_id': 'req-c519a861-dee4-475b-ac81-226e3903d765'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:09.433107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c519a861-dee4-475b-ac81-226e3903d765 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '53440d7b5288482da82c3c3f76d8e759', 'request_id': 'req-c519a861-dee4-475b-ac81-226e3903d765'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:09.449600 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c519a861-dee4-475b-ac81-226e3903d765 admin admin] The token cache is being invalidate because project 53440d7b5288482da82c3c3f76d8e759 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:09.449980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c519a861-dee4-475b-ac81-226e3903d765 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:09.451273 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c519a861-dee4-475b-ac81-226e3903d765 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:09.453177 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 931/1829] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:09 2022] DELETE /identity/v3/projects/53440d7b5288482da82c3c3f76d8e759 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:09.458832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7bb4a7-b3e3-4150-a9bc-7e2a966b1ffb None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:09.459413 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7bb4a7-b3e3-4150-a9bc-7e2a966b1ffb None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:09.459413 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7bb4a7-b3e3-4150-a9bc-7e2a966b1ffb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:09.479433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad58470-7960-4ac1-bd2d-e364268cdd57 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:09.479758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-eb7bb4a7-b3e3-4150-a9bc-7e2a966b1ffb None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:09.480186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad58470-7960-4ac1-bd2d-e364268cdd57 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:09.480465 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad58470-7960-4ac1-bd2d-e364268cdd57 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:09.480732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad58470-7960-4ac1-bd2d-e364268cdd57 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:09.481109 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-eb7bb4a7-b3e3-4150-a9bc-7e2a966b1ffb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:09.482726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-dad58470-7960-4ac1-bd2d-e364268cdd57 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:09.511435 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:09.561671 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 932/1830] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:09.563996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e1ee1c-84b6-43ac-ad0c-fb4f89562252 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:09.564562 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e1ee1c-84b6-43ac-ad0c-fb4f89562252 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:09.564876 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e1ee1c-84b6-43ac-ad0c-fb4f89562252 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:09.566979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad58470-7960-4ac1-bd2d-e364268cdd57 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:09.569209 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad58470-7960-4ac1-bd2d-e364268cdd57 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:09.580699 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 899/1831] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 236 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:09.588181 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-61e1ee1c-84b6-43ac-ad0c-fb4f89562252 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:09.589514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-61e1ee1c-84b6-43ac-ad0c-fb4f89562252 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:09.602974 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 933/1832] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:09.604364 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad58470-7960-4ac1-bd2d-e364268cdd57 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:09.615045 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad58470-7960-4ac1-bd2d-e364268cdd57 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:09.622273 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c635ed-de3d-4a77-9011-fc15d9dfe3e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:09.622955 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c635ed-de3d-4a77-9011-fc15d9dfe3e8 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:09.623237 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c635ed-de3d-4a77-9011-fc15d9dfe3e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:09.623510 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c635ed-de3d-4a77-9011-fc15d9dfe3e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:09.624192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c635ed-de3d-4a77-9011-fc15d9dfe3e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:58:09.625256 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c635ed-de3d-4a77-9011-fc15d9dfe3e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:09.627408 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c635ed-de3d-4a77-9011-fc15d9dfe3e8 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:09.647727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 900/1833] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:58:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:58:09.651855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e1ee1c-84b6-43ac-ad0c-fb4f89562252 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:09.662083 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e1ee1c-84b6-43ac-ad0c-fb4f89562252 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:09.667975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ea8b81-beb6-4105-b3dd-1bd21c424046 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:09.668724 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ea8b81-beb6-4105-b3dd-1bd21c424046 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:09.669100 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ea8b81-beb6-4105-b3dd-1bd21c424046 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:09.669472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ea8b81-beb6-4105-b3dd-1bd21c424046 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:09.670326 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ea8b81-beb6-4105-b3dd-1bd21c424046 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:09.672365 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ea8b81-beb6-4105-b3dd-1bd21c424046 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:09.705565 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 934/1834] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:58:09 2022] POST /identity/v3/projects => generated 375 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:58:09.712112 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c635ed-de3d-4a77-9011-fc15d9dfe3e8 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:09.717614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:09.723930 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c635ed-de3d-4a77-9011-fc15d9dfe3e8 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:09.806597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6eca32-3fd6-4647-8d35-29c6407ca457 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:09.807215 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6eca32-3fd6-4647-8d35-29c6407ca457 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:09.807491 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6eca32-3fd6-4647-8d35-29c6407ca457 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:09.808531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6eca32-3fd6-4647-8d35-29c6407ca457 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:09.810448 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6eca32-3fd6-4647-8d35-29c6407ca457 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:09.811235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6eca32-3fd6-4647-8d35-29c6407ca457 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:09.834575 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 901/1835] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:58:09 2022] POST /identity/v3/users => generated 288 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:58:09.839482 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ea8b81-beb6-4105-b3dd-1bd21c424046 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:09.849744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ea8b81-beb6-4105-b3dd-1bd21c424046 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:09.855036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c662b2f-74cc-4b9d-889c-c1d31caef16e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:09.855835 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c662b2f-74cc-4b9d-889c-c1d31caef16e admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:09.856179 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c662b2f-74cc-4b9d-889c-c1d31caef16e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:09.856524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c662b2f-74cc-4b9d-889c-c1d31caef16e admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:09.857240 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c662b2f-74cc-4b9d-889c-c1d31caef16e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:09.858912 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c662b2f-74cc-4b9d-889c-c1d31caef16e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:09.871509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 935/1836] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:09 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:09.876342 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6eca32-3fd6-4647-8d35-29c6407ca457 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:09.887323 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6eca32-3fd6-4647-8d35-29c6407ca457 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:09.889191 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c662b2f-74cc-4b9d-889c-c1d31caef16e admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:09.891107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5c662b2f-74cc-4b9d-889c-c1d31caef16e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:09.892866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373910c0-16ce-4479-9422-7f6177c9436a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:09.893472 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373910c0-16ce-4479-9422-7f6177c9436a admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:09.893885 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373910c0-16ce-4479-9422-7f6177c9436a admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:09.894166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373910c0-16ce-4479-9422-7f6177c9436a admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:09.894814 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373910c0-16ce-4479-9422-7f6177c9436a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:09.896514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373910c0-16ce-4479-9422-7f6177c9436a admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:09.907506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c662b2f-74cc-4b9d-889c-c1d31caef16e admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:09.911494 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 902/1837] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:09 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:09.912649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373910c0-16ce-4479-9422-7f6177c9436a admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:09.913138 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8e2b25-8d96-4ff7-9890-975eac1bf176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:09.914177 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8e2b25-8d96-4ff7-9890-975eac1bf176 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:09.914555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8e2b25-8d96-4ff7-9890-975eac1bf176 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:09.914927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8e2b25-8d96-4ff7-9890-975eac1bf176 admin admin] PATH_INFO: `/v3/projects/ba40f7e353be44918e39d72cabf4198c` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:09.917019 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8e2b25-8d96-4ff7-9890-975eac1bf176 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ba40f7e353be44918e39d72cabf4198c)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:09.919510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8e2b25-8d96-4ff7-9890-975eac1bf176 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:09.923942 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373910c0-16ce-4479-9422-7f6177c9436a admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:09.929066 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd53070c-e819-4fd2-b09d-5b90bbd1ca37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:09.929382 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd53070c-e819-4fd2-b09d-5b90bbd1ca37 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:09.930245 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd53070c-e819-4fd2-b09d-5b90bbd1ca37 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:09.930598 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd53070c-e819-4fd2-b09d-5b90bbd1ca37 admin admin] PATH_INFO: `/v3/projects/7a873f82fab44154b772d4f1e6f0a01b/users/317cab149deb4a0891ce9c37da85e037/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:09.947742 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd53070c-e819-4fd2-b09d-5b90bbd1ca37 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a873f82fab44154b772d4f1e6f0a01b, user_id=317cab149deb4a0891ce9c37da85e037, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:09.950412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd53070c-e819-4fd2-b09d-5b90bbd1ca37 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:09.963092 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 903/1838] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:09 2022] PUT /identity/v3/projects/7a873f82fab44154b772d4f1e6f0a01b/users/317cab149deb4a0891ce9c37da85e037/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:09.973172 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad2ed64-1c3a-495f-9a5c-6cf203ae4be5 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:09.973460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad2ed64-1c3a-495f-9a5c-6cf203ae4be5 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:09.973735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad2ed64-1c3a-495f-9a5c-6cf203ae4be5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:09.977933 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7b8e2b25-8d96-4ff7-9890-975eac1bf176 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ba40f7e353be44918e39d72cabf4198c', 'request_id': 'req-7b8e2b25-8d96-4ff7-9890-975eac1bf176'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:09.984527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7b8e2b25-8d96-4ff7-9890-975eac1bf176 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ba40f7e353be44918e39d72cabf4198c', 'request_id': 'req-7b8e2b25-8d96-4ff7-9890-975eac1bf176'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:09.991246 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7b8e2b25-8d96-4ff7-9890-975eac1bf176 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ba40f7e353be44918e39d72cabf4198c', 'request_id': 'req-7b8e2b25-8d96-4ff7-9890-975eac1bf176'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:10.002904 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-8ad2ed64-1c3a-495f-9a5c-6cf203ae4be5 None None] MFA Rules not processed for user `317cab149deb4a0891ce9c37da85e037`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:10.004146 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-8ad2ed64-1c3a-495f-9a5c-6cf203ae4be5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:10.005127 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7b8e2b25-8d96-4ff7-9890-975eac1bf176 admin admin] The token cache is being invalidate because project ba40f7e353be44918e39d72cabf4198c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:10.005495 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7b8e2b25-8d96-4ff7-9890-975eac1bf176 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:10.006227 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7b8e2b25-8d96-4ff7-9890-975eac1bf176 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:10.008514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 936/1839] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:09 2022] DELETE /identity/v3/projects/ba40f7e353be44918e39d72cabf4198c => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:58:10.013013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee01f12e-a87a-433b-b2b2-e1a39db4517a None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:10.013300 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee01f12e-a87a-433b-b2b2-e1a39db4517a None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:10.013580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee01f12e-a87a-433b-b2b2-e1a39db4517a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:10.032820 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-ee01f12e-a87a-433b-b2b2-e1a39db4517a None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:10.034000 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ee01f12e-a87a-433b-b2b2-e1a39db4517a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:10.090921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 904/1840] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:09 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:10.102901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad2ed64-1c3a-495f-9a5c-6cf203ae4be5 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:10.104457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-8ad2ed64-1c3a-495f-9a5c-6cf203ae4be5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:10.109053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 937/1841] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:10.116670 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410e74e4-907c-4987-8d16-3ec644581765 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:10.117489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410e74e4-907c-4987-8d16-3ec644581765 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:10.117754 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410e74e4-907c-4987-8d16-3ec644581765 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:10.139705 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-410e74e4-907c-4987-8d16-3ec644581765 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:10.140656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-410e74e4-907c-4987-8d16-3ec644581765 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:10.155783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 938/1842] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:10.157171 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410e74e4-907c-4987-8d16-3ec644581765 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:10.169170 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410e74e4-907c-4987-8d16-3ec644581765 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:10.174192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd63609d-4d15-4eb0-b8c5-447034892536 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:10.174802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd63609d-4d15-4eb0-b8c5-447034892536 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:10.175082 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd63609d-4d15-4eb0-b8c5-447034892536 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:10.175356 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd63609d-4d15-4eb0-b8c5-447034892536 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:10.176065 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd63609d-4d15-4eb0-b8c5-447034892536 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:58:10.176350 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd63609d-4d15-4eb0-b8c5-447034892536 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:10.178151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd63609d-4d15-4eb0-b8c5-447034892536 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:10.182225 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:10.185598 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:10.197272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 939/1843] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:58:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:58:10.200567 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad2ed64-1c3a-495f-9a5c-6cf203ae4be5 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:10.202027 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd63609d-4d15-4eb0-b8c5-447034892536 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:10.205998 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b577d3-4ecc-43bc-ba27-dcefd3ad2743 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:10.207353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b577d3-4ecc-43bc-ba27-dcefd3ad2743 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:10.207353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b577d3-4ecc-43bc-ba27-dcefd3ad2743 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:10.207736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b577d3-4ecc-43bc-ba27-dcefd3ad2743 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:10.215518 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd63609d-4d15-4eb0-b8c5-447034892536 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:10.220751 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b577d3-4ecc-43bc-ba27-dcefd3ad2743 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:10.222673 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b577d3-4ecc-43bc-ba27-dcefd3ad2743 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:10.224047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8571618-8563-4bd4-b5b6-ecbc3ff2889a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:10.225424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8571618-8563-4bd4-b5b6-ecbc3ff2889a admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:10.225729 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8571618-8563-4bd4-b5b6-ecbc3ff2889a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:10.226120 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8571618-8563-4bd4-b5b6-ecbc3ff2889a admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:10.227895 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8571618-8563-4bd4-b5b6-ecbc3ff2889a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:10.229848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8571618-8563-4bd4-b5b6-ecbc3ff2889a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:10.237877 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 905/1844] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:10 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:58:10.263598 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 940/1845] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:58:10 2022] POST /identity/v3/projects => generated 391 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:10.267891 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b577d3-4ecc-43bc-ba27-dcefd3ad2743 service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:10.281983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b577d3-4ecc-43bc-ba27-dcefd3ad2743 service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:10.354548 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a71c13-91ce-47bc-91d1-10f6ee3c17df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:10.355080 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a71c13-91ce-47bc-91d1-10f6ee3c17df admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:10.355359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a71c13-91ce-47bc-91d1-10f6ee3c17df admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:10.355667 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a71c13-91ce-47bc-91d1-10f6ee3c17df admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:10.356435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a71c13-91ce-47bc-91d1-10f6ee3c17df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:10.358496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a71c13-91ce-47bc-91d1-10f6ee3c17df admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:10.391002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 906/1846] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:58:10 2022] POST /identity/v3/users => generated 296 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:10.391345 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8571618-8563-4bd4-b5b6-ecbc3ff2889a admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:10.410000 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8571618-8563-4bd4-b5b6-ecbc3ff2889a admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:10.417514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439401d0-169c-44b9-bb28-f3b984e4a0a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:10.418859 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439401d0-169c-44b9-bb28-f3b984e4a0a9 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:10.419179 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439401d0-169c-44b9-bb28-f3b984e4a0a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:10.419537 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439401d0-169c-44b9-bb28-f3b984e4a0a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:10.420358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439401d0-169c-44b9-bb28-f3b984e4a0a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:10.422323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439401d0-169c-44b9-bb28-f3b984e4a0a9 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:10.434318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 941/1847] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:10 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:10.439537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a71c13-91ce-47bc-91d1-10f6ee3c17df admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:10.452371 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a71c13-91ce-47bc-91d1-10f6ee3c17df admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:10.463634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2cad041-4a17-45ce-b884-31c5871c93f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:10.464315 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cad041-4a17-45ce-b884-31c5871c93f2 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:10.464596 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cad041-4a17-45ce-b884-31c5871c93f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:10.464895 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cad041-4a17-45ce-b884-31c5871c93f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:10.469506 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2cad041-4a17-45ce-b884-31c5871c93f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:10.472311 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2cad041-4a17-45ce-b884-31c5871c93f2 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:10.486621 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 907/1848] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:10 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:10.492631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439401d0-169c-44b9-bb28-f3b984e4a0a9 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:10.504692 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439401d0-169c-44b9-bb28-f3b984e4a0a9 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:10.510405 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051c62f3-34cf-4302-be5d-ede0faf53f1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:10.511500 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051c62f3-34cf-4302-be5d-ede0faf53f1a admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:10.512027 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051c62f3-34cf-4302-be5d-ede0faf53f1a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:10.512332 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051c62f3-34cf-4302-be5d-ede0faf53f1a admin admin] PATH_INFO: `/v3/projects/e0dd7aa01e714e25851c3d287853782d/users/77721255ff1140df97cfe14f7cfdb4ff/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:10.546777 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051c62f3-34cf-4302-be5d-ede0faf53f1a admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0dd7aa01e714e25851c3d287853782d, user_id=77721255ff1140df97cfe14f7cfdb4ff, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:10.549931 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051c62f3-34cf-4302-be5d-ede0faf53f1a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:10.573999 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 942/1849] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:10 2022] PUT /identity/v3/projects/e0dd7aa01e714e25851c3d287853782d/users/77721255ff1140df97cfe14f7cfdb4ff/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:10.588244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71540c65-8d32-48ea-8116-10b6a9e3e843 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:10.589307 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71540c65-8d32-48ea-8116-10b6a9e3e843 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:10.589783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71540c65-8d32-48ea-8116-10b6a9e3e843 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:10.644188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-71540c65-8d32-48ea-8116-10b6a9e3e843 None None] MFA Rules not processed for user `77721255ff1140df97cfe14f7cfdb4ff`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:10.645654 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-71540c65-8d32-48ea-8116-10b6a9e3e843 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:10.724343 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 908/1850] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:10 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:10.736643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051c62f3-34cf-4302-be5d-ede0faf53f1a admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:10.738312 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-051c62f3-34cf-4302-be5d-ede0faf53f1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:10.795004 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051c62f3-34cf-4302-be5d-ede0faf53f1a admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:10.798537 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8d8afe-95b7-4c76-acd6-4f1cf064dc72 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:10.799153 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8d8afe-95b7-4c76-acd6-4f1cf064dc72 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:10.799410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8d8afe-95b7-4c76-acd6-4f1cf064dc72 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:10.799720 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8d8afe-95b7-4c76-acd6-4f1cf064dc72 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:10.808006 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8d8afe-95b7-4c76-acd6-4f1cf064dc72 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:10.809421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8d8afe-95b7-4c76-acd6-4f1cf064dc72 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:10.818098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 943/1851] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:10 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:13.461656 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71540c65-8d32-48ea-8116-10b6a9e3e843 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:13.463043 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-71540c65-8d32-48ea-8116-10b6a9e3e843 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:13.522260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:13.526810 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71540c65-8d32-48ea-8116-10b6a9e3e843 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:13.532346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23a4dd3-445b-48f0-9bc6-0954b4713059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:13.532976 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23a4dd3-445b-48f0-9bc6-0954b4713059 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:13.533253 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23a4dd3-445b-48f0-9bc6-0954b4713059 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:13.533521 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23a4dd3-445b-48f0-9bc6-0954b4713059 admin admin] PATH_INFO: `/v3/users/317cab149deb4a0891ce9c37da85e037` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:13.534633 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23a4dd3-445b-48f0-9bc6-0954b4713059 admin admin] RBAC: Authorizing `identity:delete_user(user_id=317cab149deb4a0891ce9c37da85e037)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:13.536645 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23a4dd3-445b-48f0-9bc6-0954b4713059 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:13.584941 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d23a4dd3-445b-48f0-9bc6-0954b4713059 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '317cab149deb4a0891ce9c37da85e037'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:13.592025 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d23a4dd3-445b-48f0-9bc6-0954b4713059 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '317cab149deb4a0891ce9c37da85e037'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:13.592698 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d23a4dd3-445b-48f0-9bc6-0954b4713059 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '317cab149deb4a0891ce9c37da85e037'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:13.593311 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d23a4dd3-445b-48f0-9bc6-0954b4713059 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '317cab149deb4a0891ce9c37da85e037'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:13.606532 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d23a4dd3-445b-48f0-9bc6-0954b4713059 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '317cab149deb4a0891ce9c37da85e037'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:13.619472 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 909/1852] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:13 2022] DELETE /identity/v3/users/317cab149deb4a0891ce9c37da85e037 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:13.624216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcedd252-c0fd-495c-a387-35afe1c335eb None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:13.624719 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcedd252-c0fd-495c-a387-35afe1c335eb None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:13.624829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcedd252-c0fd-495c-a387-35afe1c335eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:13.644165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-bcedd252-c0fd-495c-a387-35afe1c335eb None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:13.645361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-bcedd252-c0fd-495c-a387-35afe1c335eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:13.717840 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 944/1853] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:13.724881 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23a4dd3-445b-48f0-9bc6-0954b4713059 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:13.727954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d23a4dd3-445b-48f0-9bc6-0954b4713059 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:13.790363 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23a4dd3-445b-48f0-9bc6-0954b4713059 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:13.794051 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd05ea7c-dd07-4134-81c5-205501afbe6f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:13.794660 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd05ea7c-dd07-4134-81c5-205501afbe6f service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:13.794939 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd05ea7c-dd07-4134-81c5-205501afbe6f service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:13.795207 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd05ea7c-dd07-4134-81c5-205501afbe6f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:13.804618 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd05ea7c-dd07-4134-81c5-205501afbe6f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:13.806232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd05ea7c-dd07-4134-81c5-205501afbe6f service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:13.816297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 910/1854] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:14.112989 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcedd252-c0fd-495c-a387-35afe1c335eb None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:14.114524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-bcedd252-c0fd-495c-a387-35afe1c335eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:14.127190 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcedd252-c0fd-495c-a387-35afe1c335eb None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:14.141051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e167b17-8c00-466a-aa33-a8fa326b8acf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:14.141711 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e167b17-8c00-466a-aa33-a8fa326b8acf admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:14.142440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e167b17-8c00-466a-aa33-a8fa326b8acf admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:14.142440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e167b17-8c00-466a-aa33-a8fa326b8acf admin admin] PATH_INFO: `/v3/projects/7a873f82fab44154b772d4f1e6f0a01b` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:14.143513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e167b17-8c00-466a-aa33-a8fa326b8acf admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a873f82fab44154b772d4f1e6f0a01b)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:14.145451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e167b17-8c00-466a-aa33-a8fa326b8acf admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:14.185434 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:14.197329 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5e167b17-8c00-466a-aa33-a8fa326b8acf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a873f82fab44154b772d4f1e6f0a01b', 'request_id': 'req-5e167b17-8c00-466a-aa33-a8fa326b8acf'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:14.202826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5e167b17-8c00-466a-aa33-a8fa326b8acf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a873f82fab44154b772d4f1e6f0a01b', 'request_id': 'req-5e167b17-8c00-466a-aa33-a8fa326b8acf'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:14.209236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5e167b17-8c00-466a-aa33-a8fa326b8acf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a873f82fab44154b772d4f1e6f0a01b', 'request_id': 'req-5e167b17-8c00-466a-aa33-a8fa326b8acf'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:14.221884 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5e167b17-8c00-466a-aa33-a8fa326b8acf admin admin] The token cache is being invalidate because project 7a873f82fab44154b772d4f1e6f0a01b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:14.222114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5e167b17-8c00-466a-aa33-a8fa326b8acf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:14.222735 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5e167b17-8c00-466a-aa33-a8fa326b8acf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:14.224266 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 945/1855] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:14 2022] DELETE /identity/v3/projects/7a873f82fab44154b772d4f1e6f0a01b => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:14.229671 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e8f348-d115-4433-b1e2-28665cd209d0 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:14.229959 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e8f348-d115-4433-b1e2-28665cd209d0 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:14.230235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e8f348-d115-4433-b1e2-28665cd209d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:14.250495 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-02e8f348-d115-4433-b1e2-28665cd209d0 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:14.251740 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-02e8f348-d115-4433-b1e2-28665cd209d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:14.310967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 911/1856] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:14.319726 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ba287e-9519-4c6c-9939-1ef7edc7d506 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:14.321049 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ba287e-9519-4c6c-9939-1ef7edc7d506 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:14.321334 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ba287e-9519-4c6c-9939-1ef7edc7d506 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:14.343280 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-62ba287e-9519-4c6c-9939-1ef7edc7d506 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:14.344530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-62ba287e-9519-4c6c-9939-1ef7edc7d506 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:14.356374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 946/1857] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:14.360232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e8f348-d115-4433-b1e2-28665cd209d0 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:14.370599 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e8f348-d115-4433-b1e2-28665cd209d0 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:14.375189 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ac4b54-30c6-4726-b12c-7d5635885773 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:14.375821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ac4b54-30c6-4726-b12c-7d5635885773 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:14.376103 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ac4b54-30c6-4726-b12c-7d5635885773 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:14.376379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ac4b54-30c6-4726-b12c-7d5635885773 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:14.377159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ac4b54-30c6-4726-b12c-7d5635885773 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:58:14.377467 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ac4b54-30c6-4726-b12c-7d5635885773 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:14.379063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ac4b54-30c6-4726-b12c-7d5635885773 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:14.392210 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 912/1858] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:58:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:58:14.396366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ba287e-9519-4c6c-9939-1ef7edc7d506 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:14.405806 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ba287e-9519-4c6c-9939-1ef7edc7d506 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:14.410548 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ed2c3b-3cae-4c78-bd29-8e35eff19a02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:14.411215 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ed2c3b-3cae-4c78-bd29-8e35eff19a02 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:14.411581 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ed2c3b-3cae-4c78-bd29-8e35eff19a02 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:14.411930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ed2c3b-3cae-4c78-bd29-8e35eff19a02 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:14.412642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ed2c3b-3cae-4c78-bd29-8e35eff19a02 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:14.414627 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ed2c3b-3cae-4c78-bd29-8e35eff19a02 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:14.443049 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 947/1859] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:58:14 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:58:14.447993 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ac4b54-30c6-4726-b12c-7d5635885773 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:14.458534 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ac4b54-30c6-4726-b12c-7d5635885773 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:14.462177 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:14.530545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e098f07-67d4-41a2-9d14-29e054145f16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:14.531414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e098f07-67d4-41a2-9d14-29e054145f16 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:14.531730 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e098f07-67d4-41a2-9d14-29e054145f16 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:14.532003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e098f07-67d4-41a2-9d14-29e054145f16 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:14.532651 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e098f07-67d4-41a2-9d14-29e054145f16 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:14.535065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e098f07-67d4-41a2-9d14-29e054145f16 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:14.560018 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 913/1860] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:58:14 2022] POST /identity/v3/users => generated 288 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:14.563914 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ed2c3b-3cae-4c78-bd29-8e35eff19a02 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:14.574771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ed2c3b-3cae-4c78-bd29-8e35eff19a02 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:14.579474 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b346b6e1-c83e-4229-8087-b26128b16a20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:14.580130 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b346b6e1-c83e-4229-8087-b26128b16a20 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:14.580421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b346b6e1-c83e-4229-8087-b26128b16a20 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:14.580666 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b346b6e1-c83e-4229-8087-b26128b16a20 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:14.581300 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b346b6e1-c83e-4229-8087-b26128b16a20 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:14.583149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b346b6e1-c83e-4229-8087-b26128b16a20 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:14.596824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 948/1861] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:14 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:14.600954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e098f07-67d4-41a2-9d14-29e054145f16 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:14.612353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e098f07-67d4-41a2-9d14-29e054145f16 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:14.617684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e88df1f-f55b-4e2e-8f0d-560a8c87016f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:14.618229 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e88df1f-f55b-4e2e-8f0d-560a8c87016f admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:14.618508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e88df1f-f55b-4e2e-8f0d-560a8c87016f admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:14.618778 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e88df1f-f55b-4e2e-8f0d-560a8c87016f admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:14.619419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e88df1f-f55b-4e2e-8f0d-560a8c87016f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:14.621065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e88df1f-f55b-4e2e-8f0d-560a8c87016f admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:14.633351 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 914/1862] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:14 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:14.637748 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b346b6e1-c83e-4229-8087-b26128b16a20 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:14.648346 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b346b6e1-c83e-4229-8087-b26128b16a20 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:14.652994 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa90d8d5-86f5-40a2-bdfd-d13a97940d19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:14.653650 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90d8d5-86f5-40a2-bdfd-d13a97940d19 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:14.653932 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90d8d5-86f5-40a2-bdfd-d13a97940d19 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:14.654206 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90d8d5-86f5-40a2-bdfd-d13a97940d19 admin admin] PATH_INFO: `/v3/projects/683369e003c94eadbba984e7c6d5942b/users/e0f4c4aee20b4b43a02582743dbdfe34/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:14.669421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa90d8d5-86f5-40a2-bdfd-d13a97940d19 admin admin] RBAC: Authorizing `identity:create_grant(project_id=683369e003c94eadbba984e7c6d5942b, user_id=e0f4c4aee20b4b43a02582743dbdfe34, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:14.672072 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa90d8d5-86f5-40a2-bdfd-d13a97940d19 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:14.685410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 949/1863] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:14 2022] PUT /identity/v3/projects/683369e003c94eadbba984e7c6d5942b/users/e0f4c4aee20b4b43a02582743dbdfe34/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:14.696379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dda3175-20d2-40f7-b507-e27cf4e6eb62 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:14.696609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dda3175-20d2-40f7-b507-e27cf4e6eb62 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:14.696882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dda3175-20d2-40f7-b507-e27cf4e6eb62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:14.724578 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-1dda3175-20d2-40f7-b507-e27cf4e6eb62 None None] MFA Rules not processed for user `e0f4c4aee20b4b43a02582743dbdfe34`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:14.725911 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-1dda3175-20d2-40f7-b507-e27cf4e6eb62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:14.799345 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 915/1864] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:14 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:14.805648 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa90d8d5-86f5-40a2-bdfd-d13a97940d19 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:14.868200 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa90d8d5-86f5-40a2-bdfd-d13a97940d19 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:14.872963 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a876457-2cb4-4e9d-bd1e-3e4a8bae1e8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:14.873575 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a876457-2cb4-4e9d-bd1e-3e4a8bae1e8e admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:14.873878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a876457-2cb4-4e9d-bd1e-3e4a8bae1e8e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:14.874149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a876457-2cb4-4e9d-bd1e-3e4a8bae1e8e admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:14.874785 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a876457-2cb4-4e9d-bd1e-3e4a8bae1e8e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:14.876725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a876457-2cb4-4e9d-bd1e-3e4a8bae1e8e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:14.906686 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 950/1865] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:58:14 2022] POST /identity/v3/projects => generated 377 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:14.908231 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dda3175-20d2-40f7-b507-e27cf4e6eb62 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:14.919715 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dda3175-20d2-40f7-b507-e27cf4e6eb62 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:14.979465 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd89def1-c96f-4d0d-99f9-1e88ea2aa5e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:14.980092 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd89def1-c96f-4d0d-99f9-1e88ea2aa5e4 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:14.980375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd89def1-c96f-4d0d-99f9-1e88ea2aa5e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:14.980668 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd89def1-c96f-4d0d-99f9-1e88ea2aa5e4 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:14.981439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd89def1-c96f-4d0d-99f9-1e88ea2aa5e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:14.983354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd89def1-c96f-4d0d-99f9-1e88ea2aa5e4 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:15.007541 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 916/1866] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:58:14 2022] POST /identity/v3/users => generated 289 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:15.012031 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a876457-2cb4-4e9d-bd1e-3e4a8bae1e8e admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:15.022211 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a876457-2cb4-4e9d-bd1e-3e4a8bae1e8e admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:15.027418 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1fccb3-b4da-497f-914a-d54b4722c307 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:15.028132 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1fccb3-b4da-497f-914a-d54b4722c307 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:15.028412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1fccb3-b4da-497f-914a-d54b4722c307 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:15.028684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1fccb3-b4da-497f-914a-d54b4722c307 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:15.029315 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1fccb3-b4da-497f-914a-d54b4722c307 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:15.030972 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1fccb3-b4da-497f-914a-d54b4722c307 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:15.042955 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 951/1867] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:15 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:15.047053 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd89def1-c96f-4d0d-99f9-1e88ea2aa5e4 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:15.053525 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:15.058364 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd89def1-c96f-4d0d-99f9-1e88ea2aa5e4 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:15.064524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6244850-5138-4876-a8c7-9bf801b73ae0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:15.065128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6244850-5138-4876-a8c7-9bf801b73ae0 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:15.065557 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6244850-5138-4876-a8c7-9bf801b73ae0 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:15.065831 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6244850-5138-4876-a8c7-9bf801b73ae0 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:15.066475 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6244850-5138-4876-a8c7-9bf801b73ae0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:15.068119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6244850-5138-4876-a8c7-9bf801b73ae0 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:15.078218 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 917/1868] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:15 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:15.082524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1fccb3-b4da-497f-914a-d54b4722c307 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:15.092742 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1fccb3-b4da-497f-914a-d54b4722c307 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:15.097345 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff12757-67b1-4a12-ba0c-6daade6e7236 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:15.097989 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff12757-67b1-4a12-ba0c-6daade6e7236 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:15.098269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff12757-67b1-4a12-ba0c-6daade6e7236 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:15.098548 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff12757-67b1-4a12-ba0c-6daade6e7236 admin admin] PATH_INFO: `/v3/projects/04dcd139234842289ae865fbbd24eae6/users/4f73eb0883d747e18e28c9cfd20f90f8/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:15.113255 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff12757-67b1-4a12-ba0c-6daade6e7236 admin admin] RBAC: Authorizing `identity:create_grant(project_id=04dcd139234842289ae865fbbd24eae6, user_id=4f73eb0883d747e18e28c9cfd20f90f8, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:15.115937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff12757-67b1-4a12-ba0c-6daade6e7236 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:15.129091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 952/1869] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:15 2022] PUT /identity/v3/projects/04dcd139234842289ae865fbbd24eae6/users/4f73eb0883d747e18e28c9cfd20f90f8/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:15.139286 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2589008-7a4a-475f-9e3b-365537ad4507 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:15.139661 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2589008-7a4a-475f-9e3b-365537ad4507 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:15.139947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2589008-7a4a-475f-9e3b-365537ad4507 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:15.168137 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-f2589008-7a4a-475f-9e3b-365537ad4507 None None] MFA Rules not processed for user `4f73eb0883d747e18e28c9cfd20f90f8`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:15.169375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-f2589008-7a4a-475f-9e3b-365537ad4507 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:15.238208 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 918/1870] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:15 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:15.243262 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff12757-67b1-4a12-ba0c-6daade6e7236 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:15.262844 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:15.329013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff12757-67b1-4a12-ba0c-6daade6e7236 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:15.334492 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f339bf4-9265-480a-96b9-bb0717c30431 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:15.335091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f339bf4-9265-480a-96b9-bb0717c30431 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:15.335427 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f339bf4-9265-480a-96b9-bb0717c30431 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:15.335889 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f339bf4-9265-480a-96b9-bb0717c30431 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:15.336886 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f339bf4-9265-480a-96b9-bb0717c30431 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:15.338803 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f339bf4-9265-480a-96b9-bb0717c30431 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:15.372972 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 953/1871] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:58:15 2022] POST /identity/v3/projects => generated 375 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:15.376892 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2589008-7a4a-475f-9e3b-365537ad4507 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:15.388516 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2589008-7a4a-475f-9e3b-365537ad4507 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:15.446784 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6d5dbf-d8cd-4326-bde1-6c87209194a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:15.447420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6d5dbf-d8cd-4326-bde1-6c87209194a6 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:15.447733 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6d5dbf-d8cd-4326-bde1-6c87209194a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:15.448008 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6d5dbf-d8cd-4326-bde1-6c87209194a6 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:15.448661 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6d5dbf-d8cd-4326-bde1-6c87209194a6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:15.450564 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6d5dbf-d8cd-4326-bde1-6c87209194a6 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:15.472551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 919/1872] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:58:15 2022] POST /identity/v3/users => generated 294 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:15.476992 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f339bf4-9265-480a-96b9-bb0717c30431 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:15.487537 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f339bf4-9265-480a-96b9-bb0717c30431 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:15.493223 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0618efae-556e-4529-aaa4-dd8cedecbf3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:15.493754 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0618efae-556e-4529-aaa4-dd8cedecbf3f admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:15.494037 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0618efae-556e-4529-aaa4-dd8cedecbf3f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:15.494315 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0618efae-556e-4529-aaa4-dd8cedecbf3f admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:15.494955 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0618efae-556e-4529-aaa4-dd8cedecbf3f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:15.496620 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0618efae-556e-4529-aaa4-dd8cedecbf3f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:15.508408 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 954/1873] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:15 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:15.512016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6d5dbf-d8cd-4326-bde1-6c87209194a6 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:15.523731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6d5dbf-d8cd-4326-bde1-6c87209194a6 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:15.528544 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad73de5-cb3a-4e5d-a3e3-6fa7c19a145d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:15.529211 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad73de5-cb3a-4e5d-a3e3-6fa7c19a145d admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:15.529493 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad73de5-cb3a-4e5d-a3e3-6fa7c19a145d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:15.529774 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad73de5-cb3a-4e5d-a3e3-6fa7c19a145d admin admin] PATH_INFO: `/v3/projects/9ccaeef93033448e9eda0e235646e587/users/2c1f9e38b97e43e987c26b4e7f4b9db9/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:15.547155 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad73de5-cb3a-4e5d-a3e3-6fa7c19a145d admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ccaeef93033448e9eda0e235646e587, user_id=2c1f9e38b97e43e987c26b4e7f4b9db9, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:15.549958 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad73de5-cb3a-4e5d-a3e3-6fa7c19a145d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:15.564191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 920/1874] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:15 2022] PUT /identity/v3/projects/9ccaeef93033448e9eda0e235646e587/users/2c1f9e38b97e43e987c26b4e7f4b9db9/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:15.568115 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0618efae-556e-4529-aaa4-dd8cedecbf3f admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:15.578759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0618efae-556e-4529-aaa4-dd8cedecbf3f admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:15.637673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451b254d-e947-44bb-b21e-18ed95de98cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:15.638285 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451b254d-e947-44bb-b21e-18ed95de98cf admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:15.638805 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451b254d-e947-44bb-b21e-18ed95de98cf admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:15.638805 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451b254d-e947-44bb-b21e-18ed95de98cf admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:15.639453 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451b254d-e947-44bb-b21e-18ed95de98cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:15.641080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451b254d-e947-44bb-b21e-18ed95de98cf admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:15.650580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 955/1875] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:15 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:15.654688 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad73de5-cb3a-4e5d-a3e3-6fa7c19a145d admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:15.665234 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad73de5-cb3a-4e5d-a3e3-6fa7c19a145d admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:15.669837 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855f9af4-cf3c-47cd-8e61-e3992fdd6077 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:15.670470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855f9af4-cf3c-47cd-8e61-e3992fdd6077 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:15.670749 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855f9af4-cf3c-47cd-8e61-e3992fdd6077 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:15.671017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855f9af4-cf3c-47cd-8e61-e3992fdd6077 admin admin] PATH_INFO: `/v3/projects/9ccaeef93033448e9eda0e235646e587/users/2c1f9e38b97e43e987c26b4e7f4b9db9/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:15.672931 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855f9af4-cf3c-47cd-8e61-e3992fdd6077 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ccaeef93033448e9eda0e235646e587, user_id=2c1f9e38b97e43e987c26b4e7f4b9db9, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:15.675533 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855f9af4-cf3c-47cd-8e61-e3992fdd6077 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:15.686230 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 921/1876] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:15 2022] PUT /identity/v3/projects/9ccaeef93033448e9eda0e235646e587/users/2c1f9e38b97e43e987c26b4e7f4b9db9/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:15.697176 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36ca84e-803f-4b77-a5aa-369c8d814cda None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:15.697447 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36ca84e-803f-4b77-a5aa-369c8d814cda None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:15.697720 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36ca84e-803f-4b77-a5aa-369c8d814cda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:15.725144 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-e36ca84e-803f-4b77-a5aa-369c8d814cda None None] MFA Rules not processed for user `2c1f9e38b97e43e987c26b4e7f4b9db9`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:15.726825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e36ca84e-803f-4b77-a5aa-369c8d814cda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:15.791997 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 956/1877] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:15 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:15.803011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855f9af4-cf3c-47cd-8e61-e3992fdd6077 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:15.805526 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-855f9af4-cf3c-47cd-8e61-e3992fdd6077 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:15.848983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:15.870393 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855f9af4-cf3c-47cd-8e61-e3992fdd6077 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:15.874230 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a44fb30-2e12-4500-8df6-f159e99cde56 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:15.874859 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a44fb30-2e12-4500-8df6-f159e99cde56 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:15.875138 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a44fb30-2e12-4500-8df6-f159e99cde56 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:15.875512 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a44fb30-2e12-4500-8df6-f159e99cde56 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:15.936163 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a44fb30-2e12-4500-8df6-f159e99cde56 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:15.937748 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a44fb30-2e12-4500-8df6-f159e99cde56 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:15.946620 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 922/1878] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:15 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:17.613630 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36ca84e-803f-4b77-a5aa-369c8d814cda None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:17.616101 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e36ca84e-803f-4b77-a5aa-369c8d814cda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:17.628687 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36ca84e-803f-4b77-a5aa-369c8d814cda None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:17.640862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f380f2-4057-4623-8de9-b1d1f604531e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:17.641689 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f380f2-4057-4623-8de9-b1d1f604531e service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:17.642220 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f380f2-4057-4623-8de9-b1d1f604531e service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:17.642735 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f380f2-4057-4623-8de9-b1d1f604531e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:17.648949 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:17.729276 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f380f2-4057-4623-8de9-b1d1f604531e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:17.730731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f380f2-4057-4623-8de9-b1d1f604531e service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:17.739980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 957/1879] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:17 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:20.051084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a44fb30-2e12-4500-8df6-f159e99cde56 service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:20.061367 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a44fb30-2e12-4500-8df6-f159e99cde56 service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:20.065051 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ef8c67-9c0d-43e1-adba-cbc49d78b251 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:20.065674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ef8c67-9c0d-43e1-adba-cbc49d78b251 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:20.066102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ef8c67-9c0d-43e1-adba-cbc49d78b251 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:20.066386 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ef8c67-9c0d-43e1-adba-cbc49d78b251 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:20.076403 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ef8c67-9c0d-43e1-adba-cbc49d78b251 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:20.077866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ef8c67-9c0d-43e1-adba-cbc49d78b251 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:20.088495 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 923/1880] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:20 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:26.240782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f380f2-4057-4623-8de9-b1d1f604531e service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:26.390998 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f380f2-4057-4623-8de9-b1d1f604531e service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:26.395862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de28062-94aa-4dd3-9860-0bbc90116c73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:26.396395 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de28062-94aa-4dd3-9860-0bbc90116c73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:26.396674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de28062-94aa-4dd3-9860-0bbc90116c73 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:26.397010 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de28062-94aa-4dd3-9860-0bbc90116c73 admin admin] PATH_INFO: `/v3/users/e0f4c4aee20b4b43a02582743dbdfe34` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:26.398928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de28062-94aa-4dd3-9860-0bbc90116c73 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e0f4c4aee20b4b43a02582743dbdfe34)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:26.401063 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de28062-94aa-4dd3-9860-0bbc90116c73 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:26.455867 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7de28062-94aa-4dd3-9860-0bbc90116c73 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e0f4c4aee20b4b43a02582743dbdfe34'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:26.457287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7de28062-94aa-4dd3-9860-0bbc90116c73 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e0f4c4aee20b4b43a02582743dbdfe34'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:26.471833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7de28062-94aa-4dd3-9860-0bbc90116c73 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e0f4c4aee20b4b43a02582743dbdfe34'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:26.472131 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ef8c67-9c0d-43e1-adba-cbc49d78b251 service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:26.474731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-f9ef8c67-9c0d-43e1-adba-cbc49d78b251 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:26.480767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7de28062-94aa-4dd3-9860-0bbc90116c73 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e0f4c4aee20b4b43a02582743dbdfe34'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:26.491848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ef8c67-9c0d-43e1-adba-cbc49d78b251 service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:26.492540 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7de28062-94aa-4dd3-9860-0bbc90116c73 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e0f4c4aee20b4b43a02582743dbdfe34'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:26.496911 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eec0947-fe30-4c32-9aa6-2e20d3223736 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:26.497501 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eec0947-fe30-4c32-9aa6-2e20d3223736 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:26.497759 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eec0947-fe30-4c32-9aa6-2e20d3223736 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:26.498041 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eec0947-fe30-4c32-9aa6-2e20d3223736 admin admin] PATH_INFO: `/v3/users/eae1a3578c7343d39147e210369a9b46` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:26.499165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eec0947-fe30-4c32-9aa6-2e20d3223736 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eae1a3578c7343d39147e210369a9b46)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:26.500392 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 958/1881] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:26 2022] DELETE /identity/v3/users/e0f4c4aee20b4b43a02582743dbdfe34 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:26.501670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eec0947-fe30-4c32-9aa6-2e20d3223736 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:26.504489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1d252a-a673-4c58-a7de-6ac800e60a99 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:26.504799 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1d252a-a673-4c58-a7de-6ac800e60a99 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:26.505077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1d252a-a673-4c58-a7de-6ac800e60a99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:26.531817 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-ae1d252a-a673-4c58-a7de-6ac800e60a99 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:26.532918 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ae1d252a-a673-4c58-a7de-6ac800e60a99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:26.558333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:26.561498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8eec0947-fe30-4c32-9aa6-2e20d3223736 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eae1a3578c7343d39147e210369a9b46'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:26.568919 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8eec0947-fe30-4c32-9aa6-2e20d3223736 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eae1a3578c7343d39147e210369a9b46'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:26.570143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8eec0947-fe30-4c32-9aa6-2e20d3223736 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eae1a3578c7343d39147e210369a9b46'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:26.570971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8eec0947-fe30-4c32-9aa6-2e20d3223736 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eae1a3578c7343d39147e210369a9b46'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:26.586272 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8eec0947-fe30-4c32-9aa6-2e20d3223736 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eae1a3578c7343d39147e210369a9b46'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:26.601247 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 924/1882] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:26 2022] DELETE /identity/v3/users/eae1a3578c7343d39147e210369a9b46 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:26.607591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 959/1883] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:26.608603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79fbd3c-400d-4aae-a3fe-fe748492958e None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:26.608930 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79fbd3c-400d-4aae-a3fe-fe748492958e None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:26.609178 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79fbd3c-400d-4aae-a3fe-fe748492958e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:26.615187 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1d252a-a673-4c58-a7de-6ac800e60a99 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:26.617024 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ae1d252a-a673-4c58-a7de-6ac800e60a99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:26.628708 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-e79fbd3c-400d-4aae-a3fe-fe748492958e None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:26.629921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e79fbd3c-400d-4aae-a3fe-fe748492958e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:26.674780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:26.678651 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1d252a-a673-4c58-a7de-6ac800e60a99 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:26.691960 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 925/1884] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:26.702108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79fbd3c-400d-4aae-a3fe-fe748492958e None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:26.733277 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92701306-2d24-467d-a485-099b07cbb43f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:26.734050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92701306-2d24-467d-a485-099b07cbb43f service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:26.734337 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92701306-2d24-467d-a485-099b07cbb43f service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:26.734611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92701306-2d24-467d-a485-099b07cbb43f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:26.746393 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92701306-2d24-467d-a485-099b07cbb43f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:26.747972 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92701306-2d24-467d-a485-099b07cbb43f service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:26.759213 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 960/1885] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:58:26.768629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79fbd3c-400d-4aae-a3fe-fe748492958e None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:26.772862 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c3af5a-fc3d-4bb5-923b-60311bcc4182 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:26.773474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c3af5a-fc3d-4bb5-923b-60311bcc4182 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:26.773756 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c3af5a-fc3d-4bb5-923b-60311bcc4182 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:26.774024 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c3af5a-fc3d-4bb5-923b-60311bcc4182 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:26.783768 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c3af5a-fc3d-4bb5-923b-60311bcc4182 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:26.785186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c3af5a-fc3d-4bb5-923b-60311bcc4182 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:26.794992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 926/1886] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:58:27.159650 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92701306-2d24-467d-a485-099b07cbb43f service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:27.162268 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-92701306-2d24-467d-a485-099b07cbb43f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:27.187447 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92701306-2d24-467d-a485-099b07cbb43f service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:27.192855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d7406a-546a-4b83-a65c-536964b8d6b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:27.194376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d7406a-546a-4b83-a65c-536964b8d6b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:27.194376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d7406a-546a-4b83-a65c-536964b8d6b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:27.194376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d7406a-546a-4b83-a65c-536964b8d6b1 admin admin] PATH_INFO: `/v3/projects/0a03491a2ffc4855a9a18f173f81ca4b` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:27.205681 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d7406a-546a-4b83-a65c-536964b8d6b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a03491a2ffc4855a9a18f173f81ca4b)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:27.206676 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d7406a-546a-4b83-a65c-536964b8d6b1 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:27.238144 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c3af5a-fc3d-4bb5-923b-60311bcc4182 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:27.243117 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-65c3af5a-fc3d-4bb5-923b-60311bcc4182 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:27.276808 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c3af5a-fc3d-4bb5-923b-60311bcc4182 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:27.290641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da85d934-7337-4857-b982-26335f24994b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:27.291382 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da85d934-7337-4857-b982-26335f24994b admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:27.291792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da85d934-7337-4857-b982-26335f24994b admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:27.292625 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da85d934-7337-4857-b982-26335f24994b admin admin] PATH_INFO: `/v3/projects/683369e003c94eadbba984e7c6d5942b` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:27.293474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da85d934-7337-4857-b982-26335f24994b admin admin] RBAC: Authorizing `identity:delete_project(project_id=683369e003c94eadbba984e7c6d5942b)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:27.300518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da85d934-7337-4857-b982-26335f24994b admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:27.332266 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-02d7406a-546a-4b83-a65c-536964b8d6b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a03491a2ffc4855a9a18f173f81ca4b', 'request_id': 'req-02d7406a-546a-4b83-a65c-536964b8d6b1'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.351981 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-02d7406a-546a-4b83-a65c-536964b8d6b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a03491a2ffc4855a9a18f173f81ca4b', 'request_id': 'req-02d7406a-546a-4b83-a65c-536964b8d6b1'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.362421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-02d7406a-546a-4b83-a65c-536964b8d6b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a03491a2ffc4855a9a18f173f81ca4b', 'request_id': 'req-02d7406a-546a-4b83-a65c-536964b8d6b1'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.384971 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-02d7406a-546a-4b83-a65c-536964b8d6b1 admin admin] The token cache is being invalidate because project 0a03491a2ffc4855a9a18f173f81ca4b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:27.385272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-02d7406a-546a-4b83-a65c-536964b8d6b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.385935 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-02d7406a-546a-4b83-a65c-536964b8d6b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.388482 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 961/1887] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:27 2022] DELETE /identity/v3/projects/0a03491a2ffc4855a9a18f173f81ca4b => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:27.392526 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-da85d934-7337-4857-b982-26335f24994b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '683369e003c94eadbba984e7c6d5942b', 'request_id': 'req-da85d934-7337-4857-b982-26335f24994b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.392755 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d7406a-546a-4b83-a65c-536964b8d6b1 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:27.394241 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-02d7406a-546a-4b83-a65c-536964b8d6b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:27.407076 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-da85d934-7337-4857-b982-26335f24994b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '683369e003c94eadbba984e7c6d5942b', 'request_id': 'req-da85d934-7337-4857-b982-26335f24994b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.415889 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:27.421764 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-da85d934-7337-4857-b982-26335f24994b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '683369e003c94eadbba984e7c6d5942b', 'request_id': 'req-da85d934-7337-4857-b982-26335f24994b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.466374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-da85d934-7337-4857-b982-26335f24994b admin admin] The token cache is being invalidate because project 683369e003c94eadbba984e7c6d5942b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:27.466683 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-da85d934-7337-4857-b982-26335f24994b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.467400 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-da85d934-7337-4857-b982-26335f24994b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.471655 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 927/1888] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:27 2022] DELETE /identity/v3/projects/683369e003c94eadbba984e7c6d5942b => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:27.473550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da85d934-7337-4857-b982-26335f24994b admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:27.477688 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-da85d934-7337-4857-b982-26335f24994b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:27.560863 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d7406a-546a-4b83-a65c-536964b8d6b1 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:27.564552 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da85d934-7337-4857-b982-26335f24994b admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:27.569878 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62fa315-9e1a-4e74-973f-9425c84bfcf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:27.570798 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62fa315-9e1a-4e74-973f-9425c84bfcf1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:27.571184 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62fa315-9e1a-4e74-973f-9425c84bfcf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:27.571709 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62fa315-9e1a-4e74-973f-9425c84bfcf1 admin admin] PATH_INFO: `/v3/users/4f73eb0883d747e18e28c9cfd20f90f8` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:27.573151 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62fa315-9e1a-4e74-973f-9425c84bfcf1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4f73eb0883d747e18e28c9cfd20f90f8)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:27.575270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62fa315-9e1a-4e74-973f-9425c84bfcf1 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:27.575689 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb696b81-ddbf-4ffb-a60d-d3bddb2d4dc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:27.576349 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb696b81-ddbf-4ffb-a60d-d3bddb2d4dc9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:27.576628 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb696b81-ddbf-4ffb-a60d-d3bddb2d4dc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:27.576901 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb696b81-ddbf-4ffb-a60d-d3bddb2d4dc9 admin admin] PATH_INFO: `/v3/users/ea82841325a4490fada61317690a1456` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:27.578221 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb696b81-ddbf-4ffb-a60d-d3bddb2d4dc9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea82841325a4490fada61317690a1456)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:27.581920 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb696b81-ddbf-4ffb-a60d-d3bddb2d4dc9 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:27.603060 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:27.670312 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c62fa315-9e1a-4e74-973f-9425c84bfcf1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4f73eb0883d747e18e28c9cfd20f90f8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.693156 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c62fa315-9e1a-4e74-973f-9425c84bfcf1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4f73eb0883d747e18e28c9cfd20f90f8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.694435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c62fa315-9e1a-4e74-973f-9425c84bfcf1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4f73eb0883d747e18e28c9cfd20f90f8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.695753 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c62fa315-9e1a-4e74-973f-9425c84bfcf1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4f73eb0883d747e18e28c9cfd20f90f8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.708857 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-eb696b81-ddbf-4ffb-a60d-d3bddb2d4dc9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea82841325a4490fada61317690a1456'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.709588 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-eb696b81-ddbf-4ffb-a60d-d3bddb2d4dc9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea82841325a4490fada61317690a1456'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.717934 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c62fa315-9e1a-4e74-973f-9425c84bfcf1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4f73eb0883d747e18e28c9cfd20f90f8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.732009 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-eb696b81-ddbf-4ffb-a60d-d3bddb2d4dc9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea82841325a4490fada61317690a1456'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.739496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 928/1889] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:27 2022] DELETE /identity/v3/users/4f73eb0883d747e18e28c9cfd20f90f8 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:27.748344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-eb696b81-ddbf-4ffb-a60d-d3bddb2d4dc9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea82841325a4490fada61317690a1456'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.757838 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-eb696b81-ddbf-4ffb-a60d-d3bddb2d4dc9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea82841325a4490fada61317690a1456'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:27.774581 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 962/1890] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:27 2022] DELETE /identity/v3/users/ea82841325a4490fada61317690a1456 => generated 0 bytes in 381 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:28.509453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62fa315-9e1a-4e74-973f-9425c84bfcf1 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:28.510996 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-c62fa315-9e1a-4e74-973f-9425c84bfcf1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:28.565590 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb696b81-ddbf-4ffb-a60d-d3bddb2d4dc9 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:28.567250 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-eb696b81-ddbf-4ffb-a60d-d3bddb2d4dc9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:28.649116 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62fa315-9e1a-4e74-973f-9425c84bfcf1 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:28.680689 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb696b81-ddbf-4ffb-a60d-d3bddb2d4dc9 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:28.688778 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d7521-9f2c-4a3a-83ad-b2e1295cecd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:28.690238 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d7521-9f2c-4a3a-83ad-b2e1295cecd1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:28.690796 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d7521-9f2c-4a3a-83ad-b2e1295cecd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:28.691074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d7521-9f2c-4a3a-83ad-b2e1295cecd1 admin admin] PATH_INFO: `/v3/projects/69e2d5d6ea91469ea8f4562f699f3e42` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:28.694263 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857d7521-9f2c-4a3a-83ad-b2e1295cecd1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=69e2d5d6ea91469ea8f4562f699f3e42)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:28.696501 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857d7521-9f2c-4a3a-83ad-b2e1295cecd1 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:28.721684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:28.722700 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:28.722994 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:28.723269 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9 admin admin] PATH_INFO: `/v3/projects/04dcd139234842289ae865fbbd24eae6` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:28.724604 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=04dcd139234842289ae865fbbd24eae6)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:28.728156 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:28.749655 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:28.774904 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-857d7521-9f2c-4a3a-83ad-b2e1295cecd1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '69e2d5d6ea91469ea8f4562f699f3e42', 'request_id': 'req-857d7521-9f2c-4a3a-83ad-b2e1295cecd1'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:28.782361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-857d7521-9f2c-4a3a-83ad-b2e1295cecd1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '69e2d5d6ea91469ea8f4562f699f3e42', 'request_id': 'req-857d7521-9f2c-4a3a-83ad-b2e1295cecd1'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:28.786686 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:28.795876 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '04dcd139234842289ae865fbbd24eae6', 'request_id': 'req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:28.797098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-857d7521-9f2c-4a3a-83ad-b2e1295cecd1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '69e2d5d6ea91469ea8f4562f699f3e42', 'request_id': 'req-857d7521-9f2c-4a3a-83ad-b2e1295cecd1'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:28.806965 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '04dcd139234842289ae865fbbd24eae6', 'request_id': 'req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:28.813832 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '04dcd139234842289ae865fbbd24eae6', 'request_id': 'req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:28.815441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-857d7521-9f2c-4a3a-83ad-b2e1295cecd1 admin admin] The token cache is being invalidate because project 69e2d5d6ea91469ea8f4562f699f3e42 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:28.815930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-857d7521-9f2c-4a3a-83ad-b2e1295cecd1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:28.816745 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-857d7521-9f2c-4a3a-83ad-b2e1295cecd1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:28.818510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 963/1891] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:28 2022] DELETE /identity/v3/projects/69e2d5d6ea91469ea8f4562f699f3e42 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:28.825119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34128262-b400-46db-9ac7-b388d0eabae8 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:28.825418 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34128262-b400-46db-9ac7-b388d0eabae8 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:28.825665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34128262-b400-46db-9ac7-b388d0eabae8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:28.834610 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9 admin admin] The token cache is being invalidate because project 04dcd139234842289ae865fbbd24eae6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:28.834908 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:28.835673 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:28.837226 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 929/1892] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:28 2022] DELETE /identity/v3/projects/04dcd139234842289ae865fbbd24eae6 => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:28.841141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:28.843046 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:28.846947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-34128262-b400-46db-9ac7-b388d0eabae8 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:28.848180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-34128262-b400-46db-9ac7-b388d0eabae8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:28.922431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd25d712-433e-4d8c-b7a2-bc0f17bea9c9 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:28.930343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 964/1893] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:28.930504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2288d8-8781-4dd2-99ce-08fe958da3b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:28.930504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2288d8-8781-4dd2-99ce-08fe958da3b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:28.930504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2288d8-8781-4dd2-99ce-08fe958da3b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:28.930504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2288d8-8781-4dd2-99ce-08fe958da3b7 admin admin] PATH_INFO: `/v3/users/2c1f9e38b97e43e987c26b4e7f4b9db9` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:28.930979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2288d8-8781-4dd2-99ce-08fe958da3b7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c1f9e38b97e43e987c26b4e7f4b9db9)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:28.931193 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56f88af-cd3a-47b7-8c89-5e31707c441a None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:28.931479 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56f88af-cd3a-47b7-8c89-5e31707c441a None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:28.932127 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56f88af-cd3a-47b7-8c89-5e31707c441a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:28.932873 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2288d8-8781-4dd2-99ce-08fe958da3b7 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:28.957378 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-d56f88af-cd3a-47b7-8c89-5e31707c441a None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:28.958803 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d56f88af-cd3a-47b7-8c89-5e31707c441a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:28.976140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 965/1894] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:28.980504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56f88af-cd3a-47b7-8c89-5e31707c441a None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:28.988796 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-fa2288d8-8781-4dd2-99ce-08fe958da3b7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c1f9e38b97e43e987c26b4e7f4b9db9'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:28.992127 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56f88af-cd3a-47b7-8c89-5e31707c441a None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:28.998087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0503f6d6-4e52-4f34-a679-314981fdf3d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:28.998841 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0503f6d6-4e52-4f34-a679-314981fdf3d7 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:28.999097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-fa2288d8-8781-4dd2-99ce-08fe958da3b7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c1f9e38b97e43e987c26b4e7f4b9db9'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:28.999215 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0503f6d6-4e52-4f34-a679-314981fdf3d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:28.999671 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0503f6d6-4e52-4f34-a679-314981fdf3d7 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:29.000334 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0503f6d6-4e52-4f34-a679-314981fdf3d7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:58:29.000580 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-fa2288d8-8781-4dd2-99ce-08fe958da3b7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c1f9e38b97e43e987c26b4e7f4b9db9'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:29.000750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0503f6d6-4e52-4f34-a679-314981fdf3d7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:29.001505 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-fa2288d8-8781-4dd2-99ce-08fe958da3b7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c1f9e38b97e43e987c26b4e7f4b9db9'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:29.003439 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0503f6d6-4e52-4f34-a679-314981fdf3d7 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:29.017706 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-fa2288d8-8781-4dd2-99ce-08fe958da3b7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c1f9e38b97e43e987c26b4e7f4b9db9'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:29.019195 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 966/1895] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:58:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:58:29.024552 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0503f6d6-4e52-4f34-a679-314981fdf3d7 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:29.027115 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0503f6d6-4e52-4f34-a679-314981fdf3d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:29.042912 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0503f6d6-4e52-4f34-a679-314981fdf3d7 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:29.043777 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 930/1896] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:28 2022] DELETE /identity/v3/users/2c1f9e38b97e43e987c26b4e7f4b9db9 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:58:29.122486 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f21cfe-7405-41ff-83d0-6454545da1b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:29.123102 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f21cfe-7405-41ff-83d0-6454545da1b4 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:29.123376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f21cfe-7405-41ff-83d0-6454545da1b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:29.123745 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f21cfe-7405-41ff-83d0-6454545da1b4 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:29.124686 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f21cfe-7405-41ff-83d0-6454545da1b4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:29.126943 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f21cfe-7405-41ff-83d0-6454545da1b4 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:29.156000 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 967/1897] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:58:29 2022] POST /identity/v3/projects => generated 369 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:58:29.160255 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2288d8-8781-4dd2-99ce-08fe958da3b7 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:29.171445 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2288d8-8781-4dd2-99ce-08fe958da3b7 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:29.228048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:29.239905 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461116ff-78f4-46fa-8574-b67813dbd7f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:29.240516 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461116ff-78f4-46fa-8574-b67813dbd7f2 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:29.240823 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461116ff-78f4-46fa-8574-b67813dbd7f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:29.241177 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461116ff-78f4-46fa-8574-b67813dbd7f2 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:29.241832 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461116ff-78f4-46fa-8574-b67813dbd7f2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:29.243775 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461116ff-78f4-46fa-8574-b67813dbd7f2 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:29.267701 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 931/1898] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:58:29 2022] POST /identity/v3/users => generated 285 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:58:29.269032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f21cfe-7405-41ff-83d0-6454545da1b4 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:29.280867 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f21cfe-7405-41ff-83d0-6454545da1b4 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:29.286332 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ec475b-04f2-445b-9631-d27c64abab90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:29.286942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ec475b-04f2-445b-9631-d27c64abab90 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:29.287451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ec475b-04f2-445b-9631-d27c64abab90 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:29.288277 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ec475b-04f2-445b-9631-d27c64abab90 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:29.289262 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ec475b-04f2-445b-9631-d27c64abab90 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:29.290942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ec475b-04f2-445b-9631-d27c64abab90 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:29.301361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 968/1899] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:29 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:29.305398 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461116ff-78f4-46fa-8574-b67813dbd7f2 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:29.317058 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461116ff-78f4-46fa-8574-b67813dbd7f2 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:29.323059 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0e8916-43d3-44d4-8aef-c83b147b92c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:29.323718 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e8916-43d3-44d4-8aef-c83b147b92c9 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:29.323998 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e8916-43d3-44d4-8aef-c83b147b92c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:29.324361 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e8916-43d3-44d4-8aef-c83b147b92c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:29.324902 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0e8916-43d3-44d4-8aef-c83b147b92c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:29.326511 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0e8916-43d3-44d4-8aef-c83b147b92c9 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:29.329022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ec475b-04f2-445b-9631-d27c64abab90 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:29.331501 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b4ec475b-04f2-445b-9631-d27c64abab90 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:29.341232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 932/1900] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:29 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:29.345365 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0e8916-43d3-44d4-8aef-c83b147b92c9 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:29.346791 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ec475b-04f2-445b-9631-d27c64abab90 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:29.352583 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b496750d-d3af-4bb3-a098-efd57bc0aade admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:29.353227 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b496750d-d3af-4bb3-a098-efd57bc0aade admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:29.353508 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b496750d-d3af-4bb3-a098-efd57bc0aade admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:29.353783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b496750d-d3af-4bb3-a098-efd57bc0aade admin admin] PATH_INFO: `/v3/projects/9ccaeef93033448e9eda0e235646e587` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:29.355010 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b496750d-d3af-4bb3-a098-efd57bc0aade admin admin] RBAC: Authorizing `identity:delete_project(project_id=9ccaeef93033448e9eda0e235646e587)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:29.357642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b496750d-d3af-4bb3-a098-efd57bc0aade admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:29.358982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0e8916-43d3-44d4-8aef-c83b147b92c9 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:29.364565 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:29.366142 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee9386e-becf-4655-acd0-7b2ddb527adb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:29.367082 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee9386e-becf-4655-acd0-7b2ddb527adb admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:29.367389 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee9386e-becf-4655-acd0-7b2ddb527adb admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:29.367695 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee9386e-becf-4655-acd0-7b2ddb527adb admin admin] PATH_INFO: `/v3/projects/50fed3a2f4b748b9bcf141b89b4cc88a/users/ec1dc228d35942c28e1ae716a85802a1/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:29.384761 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee9386e-becf-4655-acd0-7b2ddb527adb admin admin] RBAC: Authorizing `identity:create_grant(project_id=50fed3a2f4b748b9bcf141b89b4cc88a, user_id=ec1dc228d35942c28e1ae716a85802a1, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:29.387508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee9386e-becf-4655-acd0-7b2ddb527adb admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:29.402779 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 933/1901] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:29 2022] PUT /identity/v3/projects/50fed3a2f4b748b9bcf141b89b4cc88a/users/ec1dc228d35942c28e1ae716a85802a1/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:29.415294 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d67f78-2200-4ec7-986b-12ec0e75bbad None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:29.415584 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d67f78-2200-4ec7-986b-12ec0e75bbad None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:29.416055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d67f78-2200-4ec7-986b-12ec0e75bbad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:29.416583 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b496750d-d3af-4bb3-a098-efd57bc0aade admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9ccaeef93033448e9eda0e235646e587', 'request_id': 'req-b496750d-d3af-4bb3-a098-efd57bc0aade'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:29.422544 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b496750d-d3af-4bb3-a098-efd57bc0aade admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9ccaeef93033448e9eda0e235646e587', 'request_id': 'req-b496750d-d3af-4bb3-a098-efd57bc0aade'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:29.431288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b496750d-d3af-4bb3-a098-efd57bc0aade admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9ccaeef93033448e9eda0e235646e587', 'request_id': 'req-b496750d-d3af-4bb3-a098-efd57bc0aade'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:29.449494 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b496750d-d3af-4bb3-a098-efd57bc0aade admin admin] The token cache is being invalidate because project 9ccaeef93033448e9eda0e235646e587 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:29.449789 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b496750d-d3af-4bb3-a098-efd57bc0aade admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:29.450552 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b496750d-d3af-4bb3-a098-efd57bc0aade admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:29.452310 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 969/1902] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:29 2022] DELETE /identity/v3/projects/9ccaeef93033448e9eda0e235646e587 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:58:29.455659 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-e7d67f78-2200-4ec7-986b-12ec0e75bbad None None] MFA Rules not processed for user `ec1dc228d35942c28e1ae716a85802a1`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:29.457209 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e7d67f78-2200-4ec7-986b-12ec0e75bbad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:29.541310 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 934/1903] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:29 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:29.553898 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b496750d-d3af-4bb3-a098-efd57bc0aade admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:29.556415 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b496750d-d3af-4bb3-a098-efd57bc0aade admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:29.625060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b496750d-d3af-4bb3-a098-efd57bc0aade admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:29.628984 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecda6613-29aa-4957-bc36-0f19d432964f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:29.629623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecda6613-29aa-4957-bc36-0f19d432964f service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:29.629905 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecda6613-29aa-4957-bc36-0f19d432964f service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:29.630180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecda6613-29aa-4957-bc36-0f19d432964f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:29.641862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecda6613-29aa-4957-bc36-0f19d432964f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:29.643322 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecda6613-29aa-4957-bc36-0f19d432964f service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:29.654763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 970/1904] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:29 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:29.680801 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d67f78-2200-4ec7-986b-12ec0e75bbad None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:29.684468 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e7d67f78-2200-4ec7-986b-12ec0e75bbad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:29.689259 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3675ac-a42a-4a44-9321-54a61c9e3ff6 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:29.689543 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3675ac-a42a-4a44-9321-54a61c9e3ff6 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:29.689815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3675ac-a42a-4a44-9321-54a61c9e3ff6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:29.710665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-9c3675ac-a42a-4a44-9321-54a61c9e3ff6 None None] MFA Rules not processed for user `dd1d89a64c274cefb03a68c3233cfa51`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:29.711895 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-9c3675ac-a42a-4a44-9321-54a61c9e3ff6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:29.752518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d67f78-2200-4ec7-986b-12ec0e75bbad None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:29.759441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a89f42f-72fe-4846-ab58-1c69471dc369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:29.760114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a89f42f-72fe-4846-ab58-1c69471dc369 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:29.760492 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a89f42f-72fe-4846-ab58-1c69471dc369 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:29.760820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a89f42f-72fe-4846-ab58-1c69471dc369 admin admin] PATH_INFO: `/v3/users/ec1dc228d35942c28e1ae716a85802a1` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:29.762200 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a89f42f-72fe-4846-ab58-1c69471dc369 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec1dc228d35942c28e1ae716a85802a1)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:29.764353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a89f42f-72fe-4846-ab58-1c69471dc369 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:29.776284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 971/1905] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:29 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:29.816466 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=91,keystone:INSERT=4,keystone:DELETE=5 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:29.827100 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0a89f42f-72fe-4846-ab58-1c69471dc369 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec1dc228d35942c28e1ae716a85802a1'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:29.835101 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0a89f42f-72fe-4846-ab58-1c69471dc369 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec1dc228d35942c28e1ae716a85802a1'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:29.835813 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0a89f42f-72fe-4846-ab58-1c69471dc369 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec1dc228d35942c28e1ae716a85802a1'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:29.836422 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0a89f42f-72fe-4846-ab58-1c69471dc369 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec1dc228d35942c28e1ae716a85802a1'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:29.849298 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0a89f42f-72fe-4846-ab58-1c69471dc369 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec1dc228d35942c28e1ae716a85802a1'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:29.863618 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 935/1906] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:29 2022] DELETE /identity/v3/users/ec1dc228d35942c28e1ae716a85802a1 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:58:29.868665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe23926-adab-40ec-99f2-2d89b10e997b None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:29.869636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe23926-adab-40ec-99f2-2d89b10e997b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:29.869636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe23926-adab-40ec-99f2-2d89b10e997b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:29.889432 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-dbe23926-adab-40ec-99f2-2d89b10e997b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:29.890760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-dbe23926-adab-40ec-99f2-2d89b10e997b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:29.933263 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:29.956942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 972/1907] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:29.967260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a89f42f-72fe-4846-ab58-1c69471dc369 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:29.968769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-0a89f42f-72fe-4846-ab58-1c69471dc369 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:30.030439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a89f42f-72fe-4846-ab58-1c69471dc369 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:30.034125 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d85c9a5-1cf0-40c3-a798-b3697c43c9e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:30.034722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d85c9a5-1cf0-40c3-a798-b3697c43c9e6 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:30.035000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d85c9a5-1cf0-40c3-a798-b3697c43c9e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:30.035273 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d85c9a5-1cf0-40c3-a798-b3697c43c9e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:30.044707 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d85c9a5-1cf0-40c3-a798-b3697c43c9e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:30.046232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d85c9a5-1cf0-40c3-a798-b3697c43c9e6 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:30.056315 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 936/1908] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:30.430509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe23926-adab-40ec-99f2-2d89b10e997b None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:30.432039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-dbe23926-adab-40ec-99f2-2d89b10e997b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:30.444939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe23926-adab-40ec-99f2-2d89b10e997b None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:30.449615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194f3789-b579-4054-a10a-2e2926fb42f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:30.450250 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194f3789-b579-4054-a10a-2e2926fb42f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:30.450558 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194f3789-b579-4054-a10a-2e2926fb42f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:30.450833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194f3789-b579-4054-a10a-2e2926fb42f4 admin admin] PATH_INFO: `/v3/projects/50fed3a2f4b748b9bcf141b89b4cc88a` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:30.451980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194f3789-b579-4054-a10a-2e2926fb42f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=50fed3a2f4b748b9bcf141b89b4cc88a)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:30.453900 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194f3789-b579-4054-a10a-2e2926fb42f4 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:30.502957 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-194f3789-b579-4054-a10a-2e2926fb42f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '50fed3a2f4b748b9bcf141b89b4cc88a', 'request_id': 'req-194f3789-b579-4054-a10a-2e2926fb42f4'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:30.509113 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-194f3789-b579-4054-a10a-2e2926fb42f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '50fed3a2f4b748b9bcf141b89b4cc88a', 'request_id': 'req-194f3789-b579-4054-a10a-2e2926fb42f4'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:30.515282 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-194f3789-b579-4054-a10a-2e2926fb42f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '50fed3a2f4b748b9bcf141b89b4cc88a', 'request_id': 'req-194f3789-b579-4054-a10a-2e2926fb42f4'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:30.532080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-194f3789-b579-4054-a10a-2e2926fb42f4 admin admin] The token cache is being invalidate because project 50fed3a2f4b748b9bcf141b89b4cc88a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:30.532080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-194f3789-b579-4054-a10a-2e2926fb42f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:30.532080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-194f3789-b579-4054-a10a-2e2926fb42f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:30.532080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 973/1909] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:30 2022] DELETE /identity/v3/projects/50fed3a2f4b748b9bcf141b89b4cc88a => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:30.536393 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f1d229-05dd-4738-bb06-24ddea4950df None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:30.536679 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f1d229-05dd-4738-bb06-24ddea4950df None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:30.537395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f1d229-05dd-4738-bb06-24ddea4950df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:30.558137 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-a7f1d229-05dd-4738-bb06-24ddea4950df None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:30.559336 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a7f1d229-05dd-4738-bb06-24ddea4950df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:30.603882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194f3789-b579-4054-a10a-2e2926fb42f4 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:30.605311 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-194f3789-b579-4054-a10a-2e2926fb42f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:30.630421 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 937/1910] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:30.638655 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ec7a44-66dd-4dd4-acc8-520ee697c32e None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:30.638941 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ec7a44-66dd-4dd4-acc8-520ee697c32e None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:30.639212 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ec7a44-66dd-4dd4-acc8-520ee697c32e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:30.661765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-34ec7a44-66dd-4dd4-acc8-520ee697c32e None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:30.663723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-34ec7a44-66dd-4dd4-acc8-520ee697c32e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:30.672308 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 938/1911] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:30.676221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ec7a44-66dd-4dd4-acc8-520ee697c32e None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:30.682055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194f3789-b579-4054-a10a-2e2926fb42f4 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:30.686056 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ec7a44-66dd-4dd4-acc8-520ee697c32e None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:30.686959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075cef3d-79d9-4270-b26a-6e728fd70565 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:30.687623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075cef3d-79d9-4270-b26a-6e728fd70565 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:30.687907 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075cef3d-79d9-4270-b26a-6e728fd70565 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:30.688180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075cef3d-79d9-4270-b26a-6e728fd70565 admin admin] PATH_INFO: `/v3/users/77721255ff1140df97cfe14f7cfdb4ff` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:30.689482 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075cef3d-79d9-4270-b26a-6e728fd70565 admin admin] RBAC: Authorizing `identity:delete_user(user_id=77721255ff1140df97cfe14f7cfdb4ff)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:30.691334 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075cef3d-79d9-4270-b26a-6e728fd70565 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:30.692723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80b5947-9588-4ab1-8c43-f205fb716657 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:30.693480 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80b5947-9588-4ab1-8c43-f205fb716657 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:30.694030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80b5947-9588-4ab1-8c43-f205fb716657 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:30.694460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80b5947-9588-4ab1-8c43-f205fb716657 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:30.695114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80b5947-9588-4ab1-8c43-f205fb716657 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:58:30.695419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80b5947-9588-4ab1-8c43-f205fb716657 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:30.697097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80b5947-9588-4ab1-8c43-f205fb716657 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:30.711205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 939/1912] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:58:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 14:58:30.716812 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80b5947-9588-4ab1-8c43-f205fb716657 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:30.729579 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80b5947-9588-4ab1-8c43-f205fb716657 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:30.735075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195a1a9d-2714-4d2d-b6ec-18b94376b71c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:30.735725 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a1a9d-2714-4d2d-b6ec-18b94376b71c admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:30.736003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a1a9d-2714-4d2d-b6ec-18b94376b71c admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:30.736314 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a1a9d-2714-4d2d-b6ec-18b94376b71c admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:30.736912 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195a1a9d-2714-4d2d-b6ec-18b94376b71c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:30.739027 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195a1a9d-2714-4d2d-b6ec-18b94376b71c admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:30.748134 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-075cef3d-79d9-4270-b26a-6e728fd70565 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '77721255ff1140df97cfe14f7cfdb4ff'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:30.748812 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-075cef3d-79d9-4270-b26a-6e728fd70565 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '77721255ff1140df97cfe14f7cfdb4ff'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:30.763069 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-075cef3d-79d9-4270-b26a-6e728fd70565 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '77721255ff1140df97cfe14f7cfdb4ff'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:30.765469 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:30.770689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 940/1913] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:58:30 2022] POST /identity/v3/projects => generated 383 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:30.772614 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-075cef3d-79d9-4270-b26a-6e728fd70565 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '77721255ff1140df97cfe14f7cfdb4ff'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:30.776687 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195a1a9d-2714-4d2d-b6ec-18b94376b71c admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:30.779506 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-195a1a9d-2714-4d2d-b6ec-18b94376b71c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:30.783096 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-075cef3d-79d9-4270-b26a-6e728fd70565 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '77721255ff1140df97cfe14f7cfdb4ff'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:30.791131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 974/1914] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:30 2022] DELETE /identity/v3/users/77721255ff1140df97cfe14f7cfdb4ff => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:58:30.795725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1dbe43-af4c-4cd5-b9ea-4453e74000ff None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:30.796106 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1dbe43-af4c-4cd5-b9ea-4453e74000ff None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:30.796403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1dbe43-af4c-4cd5-b9ea-4453e74000ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:30.805046 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:30.821836 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-fd1dbe43-af4c-4cd5-b9ea-4453e74000ff None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:30.823110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-fd1dbe43-af4c-4cd5-b9ea-4453e74000ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:30.861743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195a1a9d-2714-4d2d-b6ec-18b94376b71c admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:30.895452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 975/1915] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:30.907048 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1dbe43-af4c-4cd5-b9ea-4453e74000ff None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:30.908997 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-fd1dbe43-af4c-4cd5-b9ea-4453e74000ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:30.929092 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a0df89-3f9b-4277-a0ee-f01879da23c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:30.930153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a0df89-3f9b-4277-a0ee-f01879da23c8 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:30.930440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a0df89-3f9b-4277-a0ee-f01879da23c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:30.930714 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a0df89-3f9b-4277-a0ee-f01879da23c8 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:30.931782 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a0df89-3f9b-4277-a0ee-f01879da23c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:30.934419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a0df89-3f9b-4277-a0ee-f01879da23c8 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:30.964810 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 941/1916] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:58:30 2022] POST /identity/v3/users => generated 292 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:58:30.965860 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a0df89-3f9b-4277-a0ee-f01879da23c8 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:30.976071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a0df89-3f9b-4277-a0ee-f01879da23c8 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:30.976258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1dbe43-af4c-4cd5-b9ea-4453e74000ff None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:30.980827 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8684d8-251c-4aa4-9ea6-94839a39dbb3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:30.981057 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba68fb84-095e-4204-b9d9-f40e674ebb3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:30.981636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8684d8-251c-4aa4-9ea6-94839a39dbb3 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:30.981749 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba68fb84-095e-4204-b9d9-f40e674ebb3b admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:30.981977 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba68fb84-095e-4204-b9d9-f40e674ebb3b admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:30.982126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8684d8-251c-4aa4-9ea6-94839a39dbb3 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:30.982253 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba68fb84-095e-4204-b9d9-f40e674ebb3b admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:30.982514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8684d8-251c-4aa4-9ea6-94839a39dbb3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:30.982893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba68fb84-095e-4204-b9d9-f40e674ebb3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:30.984519 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba68fb84-095e-4204-b9d9-f40e674ebb3b admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:30.993130 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8684d8-251c-4aa4-9ea6-94839a39dbb3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:30.994829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8684d8-251c-4aa4-9ea6-94839a39dbb3 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:30.996290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 942/1917] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 14:58:31.001358 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba68fb84-095e-4204-b9d9-f40e674ebb3b admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:31.006976 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 976/1918] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:58:31.013520 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba68fb84-095e-4204-b9d9-f40e674ebb3b admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:31.018453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b20d184-9977-4ffc-88fc-1bc45d76db3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:31.019173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b20d184-9977-4ffc-88fc-1bc45d76db3a admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:31.019455 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b20d184-9977-4ffc-88fc-1bc45d76db3a admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:31.019792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b20d184-9977-4ffc-88fc-1bc45d76db3a admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:31.020424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b20d184-9977-4ffc-88fc-1bc45d76db3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:31.023800 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b20d184-9977-4ffc-88fc-1bc45d76db3a admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:31.035844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 943/1919] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:30 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:31.040263 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8684d8-251c-4aa4-9ea6-94839a39dbb3 service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:31.051018 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8684d8-251c-4aa4-9ea6-94839a39dbb3 service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:31.060054 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0579f4b9-9ea4-48e1-a183-615cfeb3c2a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:31.060956 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0579f4b9-9ea4-48e1-a183-615cfeb3c2a8 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:31.061393 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0579f4b9-9ea4-48e1-a183-615cfeb3c2a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:31.062076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0579f4b9-9ea4-48e1-a183-615cfeb3c2a8 admin admin] PATH_INFO: `/v3/projects/c0e37a35b902401983d17aea421212bf/users/15acaf60727346e6a9e7c8b1c982f3fd/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:31.083865 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0579f4b9-9ea4-48e1-a183-615cfeb3c2a8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c0e37a35b902401983d17aea421212bf, user_id=15acaf60727346e6a9e7c8b1c982f3fd, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:31.087814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0579f4b9-9ea4-48e1-a183-615cfeb3c2a8 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:31.104951 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 977/1920] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:31 2022] PUT /identity/v3/projects/c0e37a35b902401983d17aea421212bf/users/15acaf60727346e6a9e7c8b1c982f3fd/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:31.115263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421de3f1-4cd6-4f29-ac92-2611cb23df02 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:31.116970 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421de3f1-4cd6-4f29-ac92-2611cb23df02 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:31.116970 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421de3f1-4cd6-4f29-ac92-2611cb23df02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:31.146124 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-421de3f1-4cd6-4f29-ac92-2611cb23df02 None None] MFA Rules not processed for user `15acaf60727346e6a9e7c8b1c982f3fd`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:31.147495 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-421de3f1-4cd6-4f29-ac92-2611cb23df02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:31.223722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 944/1921] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:31 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:31.231273 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0579f4b9-9ea4-48e1-a183-615cfeb3c2a8 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:31.233580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0579f4b9-9ea4-48e1-a183-615cfeb3c2a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:31.293345 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421de3f1-4cd6-4f29-ac92-2611cb23df02 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:31.294769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-421de3f1-4cd6-4f29-ac92-2611cb23df02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:31.301040 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0579f4b9-9ea4-48e1-a183-615cfeb3c2a8 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:31.315662 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38b406d-54e4-4e7d-b779-fe083cf418d3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:31.316394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38b406d-54e4-4e7d-b779-fe083cf418d3 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:31.317599 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38b406d-54e4-4e7d-b779-fe083cf418d3 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:31.317954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38b406d-54e4-4e7d-b779-fe083cf418d3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:31.328781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38b406d-54e4-4e7d-b779-fe083cf418d3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:31.330293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38b406d-54e4-4e7d-b779-fe083cf418d3 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:31.331244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:31.340727 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 978/1922] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:31 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:58:31.383775 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421de3f1-4cd6-4f29-ac92-2611cb23df02 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:31.389314 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063e1adf-dd77-4023-97f5-005260f8a562 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:31.390013 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e1adf-dd77-4023-97f5-005260f8a562 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:31.390358 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e1adf-dd77-4023-97f5-005260f8a562 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:31.390693 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e1adf-dd77-4023-97f5-005260f8a562 admin admin] PATH_INFO: `/v3/projects/e0dd7aa01e714e25851c3d287853782d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:31.391912 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063e1adf-dd77-4023-97f5-005260f8a562 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e0dd7aa01e714e25851c3d287853782d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:31.393884 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063e1adf-dd77-4023-97f5-005260f8a562 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:31.458158 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-063e1adf-dd77-4023-97f5-005260f8a562 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e0dd7aa01e714e25851c3d287853782d', 'request_id': 'req-063e1adf-dd77-4023-97f5-005260f8a562'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:31.473061 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-063e1adf-dd77-4023-97f5-005260f8a562 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e0dd7aa01e714e25851c3d287853782d', 'request_id': 'req-063e1adf-dd77-4023-97f5-005260f8a562'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:31.483081 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-063e1adf-dd77-4023-97f5-005260f8a562 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e0dd7aa01e714e25851c3d287853782d', 'request_id': 'req-063e1adf-dd77-4023-97f5-005260f8a562'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:31.502123 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-063e1adf-dd77-4023-97f5-005260f8a562 admin admin] The token cache is being invalidate because project e0dd7aa01e714e25851c3d287853782d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:31.503306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-063e1adf-dd77-4023-97f5-005260f8a562 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:31.504433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-063e1adf-dd77-4023-97f5-005260f8a562 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:31.507180 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 945/1923] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:31 2022] DELETE /identity/v3/projects/e0dd7aa01e714e25851c3d287853782d => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 14:58:38.898886 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38b406d-54e4-4e7d-b779-fe083cf418d3 service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:38.909553 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38b406d-54e4-4e7d-b779-fe083cf418d3 service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:38.913387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:38.966996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e96ad8-8966-4857-97b9-99dba49b3870 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:38.967647 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e96ad8-8966-4857-97b9-99dba49b3870 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:38.967925 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e96ad8-8966-4857-97b9-99dba49b3870 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:38.968195 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e96ad8-8966-4857-97b9-99dba49b3870 admin admin] PATH_INFO: `/v3/users/15acaf60727346e6a9e7c8b1c982f3fd` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:38.969293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e96ad8-8966-4857-97b9-99dba49b3870 admin admin] RBAC: Authorizing `identity:delete_user(user_id=15acaf60727346e6a9e7c8b1c982f3fd)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:38.971172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e96ad8-8966-4857-97b9-99dba49b3870 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:39.016196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e9e96ad8-8966-4857-97b9-99dba49b3870 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '15acaf60727346e6a9e7c8b1c982f3fd'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:39.016837 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e9e96ad8-8966-4857-97b9-99dba49b3870 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '15acaf60727346e6a9e7c8b1c982f3fd'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:39.028958 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e9e96ad8-8966-4857-97b9-99dba49b3870 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '15acaf60727346e6a9e7c8b1c982f3fd'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:39.035779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e9e96ad8-8966-4857-97b9-99dba49b3870 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '15acaf60727346e6a9e7c8b1c982f3fd'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:39.043117 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e9e96ad8-8966-4857-97b9-99dba49b3870 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '15acaf60727346e6a9e7c8b1c982f3fd'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:39.049182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 979/1924] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:58:38 2022] DELETE /identity/v3/users/15acaf60727346e6a9e7c8b1c982f3fd => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:39.054382 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc45653-6581-4e9b-825e-d199054a6b44 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:39.054844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc45653-6581-4e9b-825e-d199054a6b44 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:39.054935 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc45653-6581-4e9b-825e-d199054a6b44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:39.075447 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-2dc45653-6581-4e9b-825e-d199054a6b44 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:39.076731 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2dc45653-6581-4e9b-825e-d199054a6b44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:39.136014 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 946/1925] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:58:39.145563 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e96ad8-8966-4857-97b9-99dba49b3870 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:39.147546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e9e96ad8-8966-4857-97b9-99dba49b3870 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:39.205168 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e96ad8-8966-4857-97b9-99dba49b3870 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:39.209487 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9f18d3-baaf-4bcc-b47e-dd46ed6f0b32 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:39.210093 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9f18d3-baaf-4bcc-b47e-dd46ed6f0b32 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:39.210376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9f18d3-baaf-4bcc-b47e-dd46ed6f0b32 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:39.210648 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9f18d3-baaf-4bcc-b47e-dd46ed6f0b32 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:39.219632 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9f18d3-baaf-4bcc-b47e-dd46ed6f0b32 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:39.221047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9f18d3-baaf-4bcc-b47e-dd46ed6f0b32 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:39.231472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 980/1926] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:39.503208 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc45653-6581-4e9b-825e-d199054a6b44 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:39.504799 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2dc45653-6581-4e9b-825e-d199054a6b44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:39.516924 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc45653-6581-4e9b-825e-d199054a6b44 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:39.529588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26fbf67-9e2b-4cb7-b724-b15ed777bdf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:39.530005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26fbf67-9e2b-4cb7-b724-b15ed777bdf6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:39.530299 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26fbf67-9e2b-4cb7-b724-b15ed777bdf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:39.530597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26fbf67-9e2b-4cb7-b724-b15ed777bdf6 admin admin] PATH_INFO: `/v3/projects/c0e37a35b902401983d17aea421212bf` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:39.531981 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26fbf67-9e2b-4cb7-b724-b15ed777bdf6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c0e37a35b902401983d17aea421212bf)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:39.533890 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26fbf67-9e2b-4cb7-b724-b15ed777bdf6 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:39.578899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:39.582431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a26fbf67-9e2b-4cb7-b724-b15ed777bdf6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c0e37a35b902401983d17aea421212bf', 'request_id': 'req-a26fbf67-9e2b-4cb7-b724-b15ed777bdf6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:39.590859 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a26fbf67-9e2b-4cb7-b724-b15ed777bdf6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c0e37a35b902401983d17aea421212bf', 'request_id': 'req-a26fbf67-9e2b-4cb7-b724-b15ed777bdf6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:39.596412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a26fbf67-9e2b-4cb7-b724-b15ed777bdf6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c0e37a35b902401983d17aea421212bf', 'request_id': 'req-a26fbf67-9e2b-4cb7-b724-b15ed777bdf6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:39.607944 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a26fbf67-9e2b-4cb7-b724-b15ed777bdf6 admin admin] The token cache is being invalidate because project c0e37a35b902401983d17aea421212bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:58:39.608232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a26fbf67-9e2b-4cb7-b724-b15ed777bdf6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:39.608861 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a26fbf67-9e2b-4cb7-b724-b15ed777bdf6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:58:39.610335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 947/1927] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:58:39 2022] DELETE /identity/v3/projects/c0e37a35b902401983d17aea421212bf => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:39.615150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b309bc1e-875d-40f7-81cf-a02ab3ba499e None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:39.615433 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b309bc1e-875d-40f7-81cf-a02ab3ba499e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:39.615745 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b309bc1e-875d-40f7-81cf-a02ab3ba499e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:39.635833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-b309bc1e-875d-40f7-81cf-a02ab3ba499e None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:39.637024 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b309bc1e-875d-40f7-81cf-a02ab3ba499e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:39.661094 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:39.697931 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 981/1928] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:39.706979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99aa9d1c-34ee-4790-bdb7-16b7e97a6c22 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:39.707264 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99aa9d1c-34ee-4790-bdb7-16b7e97a6c22 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:39.707533 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99aa9d1c-34ee-4790-bdb7-16b7e97a6c22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:39.728425 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-99aa9d1c-34ee-4790-bdb7-16b7e97a6c22 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:39.729635 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-99aa9d1c-34ee-4790-bdb7-16b7e97a6c22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:39.737616 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 948/1929] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:39.742034 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b309bc1e-875d-40f7-81cf-a02ab3ba499e None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:39.751670 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b309bc1e-875d-40f7-81cf-a02ab3ba499e None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:39.756519 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7f48f4-60bd-4759-ae1b-92055105b5cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:39.757130 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7f48f4-60bd-4759-ae1b-92055105b5cc admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:39.757410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7f48f4-60bd-4759-ae1b-92055105b5cc admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:39.757686 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7f48f4-60bd-4759-ae1b-92055105b5cc admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:39.758337 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7f48f4-60bd-4759-ae1b-92055105b5cc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:58:39.758636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7f48f4-60bd-4759-ae1b-92055105b5cc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:39.760274 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7f48f4-60bd-4759-ae1b-92055105b5cc admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:39.772714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 982/1930] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:58:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:58:39.776935 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99aa9d1c-34ee-4790-bdb7-16b7e97a6c22 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:39.786515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99aa9d1c-34ee-4790-bdb7-16b7e97a6c22 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:39.792166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a5c385-9472-4e3e-803f-ccc4f3ca8e51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:39.792831 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a5c385-9472-4e3e-803f-ccc4f3ca8e51 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:39.793171 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a5c385-9472-4e3e-803f-ccc4f3ca8e51 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:39.793507 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a5c385-9472-4e3e-803f-ccc4f3ca8e51 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:39.794204 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a5c385-9472-4e3e-803f-ccc4f3ca8e51 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:39.796222 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a5c385-9472-4e3e-803f-ccc4f3ca8e51 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:39.828648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 949/1931] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:58:39 2022] POST /identity/v3/projects => generated 373 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:39.829840 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7f48f4-60bd-4759-ae1b-92055105b5cc admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:39.844885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7f48f4-60bd-4759-ae1b-92055105b5cc admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:39.906333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1081b64a-dc8b-4c7f-92eb-470084802e58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:39.906999 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1081b64a-dc8b-4c7f-92eb-470084802e58 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:39.907319 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1081b64a-dc8b-4c7f-92eb-470084802e58 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:39.907694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1081b64a-dc8b-4c7f-92eb-470084802e58 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:39.908323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1081b64a-dc8b-4c7f-92eb-470084802e58 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:39.910223 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1081b64a-dc8b-4c7f-92eb-470084802e58 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:39.934615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 983/1932] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:58:39 2022] POST /identity/v3/users => generated 287 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:58:39.936082 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a5c385-9472-4e3e-803f-ccc4f3ca8e51 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:39.948020 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a5c385-9472-4e3e-803f-ccc4f3ca8e51 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:39.952732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a5135d-7125-4c53-9ae6-950b3cc593f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:39.953359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a5135d-7125-4c53-9ae6-950b3cc593f5 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:39.953635 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a5135d-7125-4c53-9ae6-950b3cc593f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:39.953901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a5135d-7125-4c53-9ae6-950b3cc593f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:39.954571 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a5135d-7125-4c53-9ae6-950b3cc593f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:39.956462 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a5135d-7125-4c53-9ae6-950b3cc593f5 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:39.968154 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 950/1933] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:39 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:39.972278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1081b64a-dc8b-4c7f-92eb-470084802e58 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:39.982316 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1081b64a-dc8b-4c7f-92eb-470084802e58 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:39.987271 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a179e9-73fd-4436-8898-6b43b0902626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:39.987976 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a179e9-73fd-4436-8898-6b43b0902626 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:39.988273 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a179e9-73fd-4436-8898-6b43b0902626 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:39.988543 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a179e9-73fd-4436-8898-6b43b0902626 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:39.989185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a179e9-73fd-4436-8898-6b43b0902626 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:39.990782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a179e9-73fd-4436-8898-6b43b0902626 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:40.001348 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 984/1934] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:58:39 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:58:40.005378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a5135d-7125-4c53-9ae6-950b3cc593f5 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:40.015846 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a5135d-7125-4c53-9ae6-950b3cc593f5 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:40.020574 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d12903-81d8-4991-8f4a-1bf379e7fccd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:40.021239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d12903-81d8-4991-8f4a-1bf379e7fccd admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:40.022128 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d12903-81d8-4991-8f4a-1bf379e7fccd admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:40.022405 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d12903-81d8-4991-8f4a-1bf379e7fccd admin admin] PATH_INFO: `/v3/projects/0d647ac8f50049dab664e81a70e106ca/users/d2da2ba5ff4a449ba678db76180c4cbd/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:40.038005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d12903-81d8-4991-8f4a-1bf379e7fccd admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d647ac8f50049dab664e81a70e106ca, user_id=d2da2ba5ff4a449ba678db76180c4cbd, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:40.040852 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d12903-81d8-4991-8f4a-1bf379e7fccd admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:40.053975 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 951/1935] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:58:40 2022] PUT /identity/v3/projects/0d647ac8f50049dab664e81a70e106ca/users/d2da2ba5ff4a449ba678db76180c4cbd/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:58:40.064209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1076c27e-97f5-457e-9ae7-259288c61703 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:40.064493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1076c27e-97f5-457e-9ae7-259288c61703 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:40.064791 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1076c27e-97f5-457e-9ae7-259288c61703 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:40.093916 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-1076c27e-97f5-457e-9ae7-259288c61703 None None] MFA Rules not processed for user `d2da2ba5ff4a449ba678db76180c4cbd`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:40.095033 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1076c27e-97f5-457e-9ae7-259288c61703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:40.157735 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 985/1936] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:40 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:40.168947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d12903-81d8-4991-8f4a-1bf379e7fccd admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:40.170654 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-35d12903-81d8-4991-8f4a-1bf379e7fccd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:40.227834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d12903-81d8-4991-8f4a-1bf379e7fccd admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:40.231456 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d408b9-01d8-4240-ad51-f54cddaae864 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:40.232084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d408b9-01d8-4240-ad51-f54cddaae864 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:40.232349 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d408b9-01d8-4240-ad51-f54cddaae864 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:40.232646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d408b9-01d8-4240-ad51-f54cddaae864 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:40.241699 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d408b9-01d8-4240-ad51-f54cddaae864 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:40.243185 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d408b9-01d8-4240-ad51-f54cddaae864 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:40.252174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 952/1937] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:58:40 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:58:41.726709 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f391087-0455-43d2-804f-14258dd819b7 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:41.727225 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f391087-0455-43d2-804f-14258dd819b7 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:41.727314 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f391087-0455-43d2-804f-14258dd819b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:41.746412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-4f391087-0455-43d2-804f-14258dd819b7 None None] MFA Rules not processed for user `d2da2ba5ff4a449ba678db76180c4cbd`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:41.747664 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-4f391087-0455-43d2-804f-14258dd819b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:41.754893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 986/1938] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:58:41 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:50.248250 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:DELETE=4,keystone:SELECT=60,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:51.740757 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:58:54.747937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ed5bb4-ba52-4046-a71b-dde76c67ee25 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:54.748663 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ed5bb4-ba52-4046-a71b-dde76c67ee25 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:54.748663 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ed5bb4-ba52-4046-a71b-dde76c67ee25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:54.769701 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-17ed5bb4-ba52-4046-a71b-dde76c67ee25 None None] MFA Rules not processed for user `e0d0f8598fac40449bcb2a55ff8df8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:58:54.771325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-17ed5bb4-ba52-4046-a71b-dde76c67ee25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:54.843538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 953/1939] 10.222.0.36 () {58 vars in 1009 bytes} [Mon Oct 31 14:58:54 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:58:54.890934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f391087-0455-43d2-804f-14258dd819b7 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:58:54.892412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-4f391087-0455-43d2-804f-14258dd819b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:58:54.984448 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f391087-0455-43d2-804f-14258dd819b7 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:58:54.989376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed9aad3-87f1-49d6-bb4d-77e07015fb84 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4ae5e20cc5b40ccbdb951f908f5080d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:58:54.990709 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed9aad3-87f1-49d6-bb4d-77e07015fb84 service swift] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:58:54.990992 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed9aad3-87f1-49d6-bb4d-77e07015fb84 service swift] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:58:54.991264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed9aad3-87f1-49d6-bb4d-77e07015fb84 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:58:55.002416 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed9aad3-87f1-49d6-bb4d-77e07015fb84 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:58:55.004070 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed9aad3-87f1-49d6-bb4d-77e07015fb84 service swift] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:58:55.013798 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 987/1940] 10.222.0.36 () {60 vars in 1384 bytes} [Mon Oct 31 14:58:54 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 31 14:59:00.545341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c47520a-184a-42bc-8785-971be902da2c None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:00.545947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c47520a-184a-42bc-8785-971be902da2c None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:00.545947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c47520a-184a-42bc-8785-971be902da2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:00.585658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-1c47520a-184a-42bc-8785-971be902da2c None None] MFA Rules not processed for user `e0d0f8598fac40449bcb2a55ff8df8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:00.586907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-1c47520a-184a-42bc-8785-971be902da2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:00.623822 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 954/1941] 10.222.0.36 () {58 vars in 1009 bytes} [Mon Oct 31 14:59:00 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:59:00.627489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed9aad3-87f1-49d6-bb4d-77e07015fb84 service swift] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:00.642774 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed9aad3-87f1-49d6-bb4d-77e07015fb84 service swift] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:00.657846 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a797f0-3a5e-49d2-8a89-1522f91cec46 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4ae5e20cc5b40ccbdb951f908f5080d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:00.659139 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a797f0-3a5e-49d2-8a89-1522f91cec46 service swift] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:00.660300 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a797f0-3a5e-49d2-8a89-1522f91cec46 service swift] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:00.660682 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a797f0-3a5e-49d2-8a89-1522f91cec46 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:00.684105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a797f0-3a5e-49d2-8a89-1522f91cec46 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:00.685778 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a797f0-3a5e-49d2-8a89-1522f91cec46 service swift] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:00.706963 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 988/1942] 10.222.0.36 () {60 vars in 1384 bytes} [Mon Oct 31 14:59:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 31 14:59:02.132254 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d993c36-60dd-42d9-9871-a72e8f5b2455 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:02.132794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d993c36-60dd-42d9-9871-a72e8f5b2455 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:02.132794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d993c36-60dd-42d9-9871-a72e8f5b2455 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:02.196349 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-2d993c36-60dd-42d9-9871-a72e8f5b2455 None None] MFA Rules not processed for user `e6bfe2c1ba054935ae8e9fcc054faf68`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:02.197602 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2d993c36-60dd-42d9-9871-a72e8f5b2455 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:02.383032 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 955/1943] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:59:02 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:59:08.844688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2220ec3f-ab5b-4889-9101-965cf9569724 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:08.845697 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2220ec3f-ab5b-4889-9101-965cf9569724 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:08.846139 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2220ec3f-ab5b-4889-9101-965cf9569724 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:08.903775 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-2220ec3f-ab5b-4889-9101-965cf9569724 None None] MFA Rules not processed for user `cdcbcb2ed9e44dbe8444e531fc1eb707`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:08.905452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2220ec3f-ab5b-4889-9101-965cf9569724 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:09.043394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 989/1944] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:59:08 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:59:12.370663 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=53 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:59:13.283398 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ae0754-f559-459a-88ec-c3e311989ec1 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:13.283398 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ae0754-f559-459a-88ec-c3e311989ec1 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:13.283758 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ae0754-f559-459a-88ec-c3e311989ec1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:13.311678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-50ae0754-f559-459a-88ec-c3e311989ec1 None None] MFA Rules not processed for user `e0d0f8598fac40449bcb2a55ff8df8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:13.312908 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-50ae0754-f559-459a-88ec-c3e311989ec1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:13.321641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 956/1945] 10.222.0.36 () {58 vars in 1009 bytes} [Mon Oct 31 14:59:13 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:59:13.346492 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2220ec3f-ab5b-4889-9101-965cf9569724 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:13.362420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2220ec3f-ab5b-4889-9101-965cf9569724 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:13.368161 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ae6891-35bf-4224-911c-4b6619122488 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4ae5e20cc5b40ccbdb951f908f5080d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:13.368849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ae6891-35bf-4224-911c-4b6619122488 service swift] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:13.369131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ae6891-35bf-4224-911c-4b6619122488 service swift] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:13.369400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ae6891-35bf-4224-911c-4b6619122488 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:13.378110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ae6891-35bf-4224-911c-4b6619122488 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:13.380423 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ae6891-35bf-4224-911c-4b6619122488 service swift] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:13.391853 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 990/1946] 10.222.0.36 () {60 vars in 1384 bytes} [Mon Oct 31 14:59:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 31 14:59:14.267810 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3c7265-26b5-4da2-940c-29f9415f4996 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:14.268301 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3c7265-26b5-4da2-940c-29f9415f4996 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:14.268424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3c7265-26b5-4da2-940c-29f9415f4996 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:14.293892 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-ae3c7265-26b5-4da2-940c-29f9415f4996 None None] MFA Rules not processed for user `e0d0f8598fac40449bcb2a55ff8df8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:14.295817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ae3c7265-26b5-4da2-940c-29f9415f4996 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:14.313506 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 957/1947] 10.222.0.36 () {58 vars in 967 bytes} [Mon Oct 31 14:59:14 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:59:14.321424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ae6891-35bf-4224-911c-4b6619122488 service swift] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:14.335035 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ae6891-35bf-4224-911c-4b6619122488 service swift] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:14.344120 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e292f9ef-ead6-4424-9cf3-0340d8136d81 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4ae5e20cc5b40ccbdb951f908f5080d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:14.346229 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e292f9ef-ead6-4424-9cf3-0340d8136d81 service swift] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:14.346418 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e292f9ef-ead6-4424-9cf3-0340d8136d81 service swift] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:14.346722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e292f9ef-ead6-4424-9cf3-0340d8136d81 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:14.370398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e292f9ef-ead6-4424-9cf3-0340d8136d81 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:14.375623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e292f9ef-ead6-4424-9cf3-0340d8136d81 service swift] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:14.406643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 991/1948] 10.222.0.36 () {60 vars in 1384 bytes} [Mon Oct 31 14:59:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 31 14:59:14.761514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9d0e90-60c6-408f-bcc9-46e6de698b20 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:14.761796 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9d0e90-60c6-408f-bcc9-46e6de698b20 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:14.762074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9d0e90-60c6-408f-bcc9-46e6de698b20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:14.817591 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-3e9d0e90-60c6-408f-bcc9-46e6de698b20 None None] MFA Rules not processed for user `e0d0f8598fac40449bcb2a55ff8df8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:14.818853 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-3e9d0e90-60c6-408f-bcc9-46e6de698b20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:14.830147 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 958/1949] 10.222.0.36 () {58 vars in 1009 bytes} [Mon Oct 31 14:59:14 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:59:14.846245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e292f9ef-ead6-4424-9cf3-0340d8136d81 service swift] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:14.870591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e292f9ef-ead6-4424-9cf3-0340d8136d81 service swift] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:14.881420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e0d4b1-0d73-4ea5-9b95-84862c23cd6f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4ae5e20cc5b40ccbdb951f908f5080d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:14.886790 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e0d4b1-0d73-4ea5-9b95-84862c23cd6f service swift] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:14.887244 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e0d4b1-0d73-4ea5-9b95-84862c23cd6f service swift] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:14.887536 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e0d4b1-0d73-4ea5-9b95-84862c23cd6f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:14.909560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e0d4b1-0d73-4ea5-9b95-84862c23cd6f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:14.911487 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e0d4b1-0d73-4ea5-9b95-84862c23cd6f service swift] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:14.943876 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 992/1950] 10.222.0.36 () {60 vars in 1384 bytes} [Mon Oct 31 14:59:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 31 14:59:15.286230 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689b50a7-80bf-4a97-9f31-f361a6f7131d None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:15.286774 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689b50a7-80bf-4a97-9f31-f361a6f7131d None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:15.286774 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689b50a7-80bf-4a97-9f31-f361a6f7131d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:15.340453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-689b50a7-80bf-4a97-9f31-f361a6f7131d None None] MFA Rules not processed for user `e0d0f8598fac40449bcb2a55ff8df8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:15.352107 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-689b50a7-80bf-4a97-9f31-f361a6f7131d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:15.372838 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 959/1951] 10.222.0.36 () {58 vars in 967 bytes} [Mon Oct 31 14:59:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:59:15.396169 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e0d4b1-0d73-4ea5-9b95-84862c23cd6f service swift] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:15.411085 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e0d4b1-0d73-4ea5-9b95-84862c23cd6f service swift] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:15.417985 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c51c062-55db-4db9-b10e-9e6530b31b83 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4ae5e20cc5b40ccbdb951f908f5080d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:15.418767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c51c062-55db-4db9-b10e-9e6530b31b83 service swift] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:15.419051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c51c062-55db-4db9-b10e-9e6530b31b83 service swift] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:15.419364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c51c062-55db-4db9-b10e-9e6530b31b83 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:15.432895 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c51c062-55db-4db9-b10e-9e6530b31b83 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:15.434592 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c51c062-55db-4db9-b10e-9e6530b31b83 service swift] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:15.444651 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 993/1952] 10.222.0.36 () {60 vars in 1384 bytes} [Mon Oct 31 14:59:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 31 14:59:15.492925 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ce8e18-1023-451c-b674-63ef95c19d4d None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:15.493219 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ce8e18-1023-451c-b674-63ef95c19d4d None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:15.493501 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ce8e18-1023-451c-b674-63ef95c19d4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:15.515285 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-d3ce8e18-1023-451c-b674-63ef95c19d4d None None] MFA Rules not processed for user `e0d0f8598fac40449bcb2a55ff8df8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:15.516438 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d3ce8e18-1023-451c-b674-63ef95c19d4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:15.525550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 960/1953] 10.222.0.36 () {58 vars in 967 bytes} [Mon Oct 31 14:59:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:59:15.539671 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c51c062-55db-4db9-b10e-9e6530b31b83 service swift] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:15.552917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c51c062-55db-4db9-b10e-9e6530b31b83 service swift] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:15.559960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a323266-7664-41f7-b790-01dd8a8be8b9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4ae5e20cc5b40ccbdb951f908f5080d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:15.561444 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a323266-7664-41f7-b790-01dd8a8be8b9 service swift] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:15.561735 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a323266-7664-41f7-b790-01dd8a8be8b9 service swift] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:15.562008 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a323266-7664-41f7-b790-01dd8a8be8b9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:15.572970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a323266-7664-41f7-b790-01dd8a8be8b9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:15.574464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a323266-7664-41f7-b790-01dd8a8be8b9 service swift] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:15.584795 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 994/1954] 10.222.0.36 () {60 vars in 1384 bytes} [Mon Oct 31 14:59:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 31 14:59:17.618673 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ce8e18-1023-451c-b674-63ef95c19d4d None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:17.620596 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d3ce8e18-1023-451c-b674-63ef95c19d4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:17.693608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ce8e18-1023-451c-b674-63ef95c19d4d None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:17.698766 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03bbd697-a8f3-4ad6-bdbe-babf0635fbd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:17.699622 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bbd697-a8f3-4ad6-bdbe-babf0635fbd4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:17.699910 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bbd697-a8f3-4ad6-bdbe-babf0635fbd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:17.700183 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bbd697-a8f3-4ad6-bdbe-babf0635fbd4 admin admin] PATH_INFO: `/v3/users/dd1d89a64c274cefb03a68c3233cfa51` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:17.701462 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03bbd697-a8f3-4ad6-bdbe-babf0635fbd4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dd1d89a64c274cefb03a68c3233cfa51)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:17.703436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03bbd697-a8f3-4ad6-bdbe-babf0635fbd4 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:17.763168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-03bbd697-a8f3-4ad6-bdbe-babf0635fbd4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dd1d89a64c274cefb03a68c3233cfa51'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:17.775887 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-03bbd697-a8f3-4ad6-bdbe-babf0635fbd4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dd1d89a64c274cefb03a68c3233cfa51'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:17.777476 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a323266-7664-41f7-b790-01dd8a8be8b9 service swift] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:17.778970 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-03bbd697-a8f3-4ad6-bdbe-babf0635fbd4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dd1d89a64c274cefb03a68c3233cfa51'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:17.785773 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-03bbd697-a8f3-4ad6-bdbe-babf0635fbd4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dd1d89a64c274cefb03a68c3233cfa51'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:17.791912 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a323266-7664-41f7-b790-01dd8a8be8b9 service swift] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:17.798016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed02690-e382-4f87-a59a-131e263e1569 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:17.798901 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed02690-e382-4f87-a59a-131e263e1569 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:17.799393 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed02690-e382-4f87-a59a-131e263e1569 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:17.799908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed02690-e382-4f87-a59a-131e263e1569 admin admin] PATH_INFO: `/v3/users/d2da2ba5ff4a449ba678db76180c4cbd` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:17.801407 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed02690-e382-4f87-a59a-131e263e1569 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d2da2ba5ff4a449ba678db76180c4cbd)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:17.804246 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed02690-e382-4f87-a59a-131e263e1569 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:17.805213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-03bbd697-a8f3-4ad6-bdbe-babf0635fbd4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dd1d89a64c274cefb03a68c3233cfa51'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:17.820498 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:59:17.828417 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 961/1955] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:59:17 2022] DELETE /identity/v3/users/dd1d89a64c274cefb03a68c3233cfa51 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:59:17.872968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-3ed02690-e382-4f87-a59a-131e263e1569 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd2da2ba5ff4a449ba678db76180c4cbd'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:17.873693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-3ed02690-e382-4f87-a59a-131e263e1569 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd2da2ba5ff4a449ba678db76180c4cbd'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:17.890076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-3ed02690-e382-4f87-a59a-131e263e1569 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd2da2ba5ff4a449ba678db76180c4cbd'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:17.904448 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-3ed02690-e382-4f87-a59a-131e263e1569 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd2da2ba5ff4a449ba678db76180c4cbd'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:17.920805 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-3ed02690-e382-4f87-a59a-131e263e1569 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd2da2ba5ff4a449ba678db76180c4cbd'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:17.937479 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 995/1956] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:59:17 2022] DELETE /identity/v3/users/d2da2ba5ff4a449ba678db76180c4cbd => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:59:17.942133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadd4acd-3f6c-4e10-a19d-0ff83b672a29 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:17.942642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadd4acd-3f6c-4e10-a19d-0ff83b672a29 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:17.943073 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadd4acd-3f6c-4e10-a19d-0ff83b672a29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:17.980786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-cadd4acd-3f6c-4e10-a19d-0ff83b672a29 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:17.982452 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-cadd4acd-3f6c-4e10-a19d-0ff83b672a29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:18.042126 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:59:18.083597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 962/1957] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:59:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:59:18.093815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed02690-e382-4f87-a59a-131e263e1569 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:18.097277 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-3ed02690-e382-4f87-a59a-131e263e1569 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:18.199860 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed02690-e382-4f87-a59a-131e263e1569 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:18.209632 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d46e3c0-968b-46d2-860b-29c851a44fdd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:18.210504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d46e3c0-968b-46d2-860b-29c851a44fdd service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:18.211328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d46e3c0-968b-46d2-860b-29c851a44fdd service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:18.211654 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d46e3c0-968b-46d2-860b-29c851a44fdd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:18.225077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d46e3c0-968b-46d2-860b-29c851a44fdd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:18.226907 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d46e3c0-968b-46d2-860b-29c851a44fdd service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:18.232771 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadd4acd-3f6c-4e10-a19d-0ff83b672a29 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:18.234413 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-cadd4acd-3f6c-4e10-a19d-0ff83b672a29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:18.250327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 996/1958] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:59:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:59:18.280307 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadd4acd-3f6c-4e10-a19d-0ff83b672a29 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:18.294045 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd616eb8-7846-4193-b85b-62d32707696d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:18.294912 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd616eb8-7846-4193-b85b-62d32707696d admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:18.295234 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd616eb8-7846-4193-b85b-62d32707696d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:18.296831 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd616eb8-7846-4193-b85b-62d32707696d admin admin] PATH_INFO: `/v3/projects/4075982a0ef04a6a8edbe37fde6cb657` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:18.299429 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd616eb8-7846-4193-b85b-62d32707696d admin admin] RBAC: Authorizing `identity:delete_project(project_id=4075982a0ef04a6a8edbe37fde6cb657)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:18.302228 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd616eb8-7846-4193-b85b-62d32707696d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:18.530530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d46e3c0-968b-46d2-860b-29c851a44fdd service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:18.531733 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-fd616eb8-7846-4193-b85b-62d32707696d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4075982a0ef04a6a8edbe37fde6cb657', 'request_id': 'req-fd616eb8-7846-4193-b85b-62d32707696d'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:18.540440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-4d46e3c0-968b-46d2-860b-29c851a44fdd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:18.547357 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-fd616eb8-7846-4193-b85b-62d32707696d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4075982a0ef04a6a8edbe37fde6cb657', 'request_id': 'req-fd616eb8-7846-4193-b85b-62d32707696d'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:18.570886 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d46e3c0-968b-46d2-860b-29c851a44fdd service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:18.577106 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-fd616eb8-7846-4193-b85b-62d32707696d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4075982a0ef04a6a8edbe37fde6cb657', 'request_id': 'req-fd616eb8-7846-4193-b85b-62d32707696d'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:18.622815 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-fd616eb8-7846-4193-b85b-62d32707696d admin admin] The token cache is being invalidate because project 4075982a0ef04a6a8edbe37fde6cb657 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:59:18.624247 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-fd616eb8-7846-4193-b85b-62d32707696d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:18.625890 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-fd616eb8-7846-4193-b85b-62d32707696d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:18.627966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 963/1959] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:59:18 2022] DELETE /identity/v3/projects/4075982a0ef04a6a8edbe37fde6cb657 => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:59:18.664618 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a5e8fa-dcd1-4ef7-ba78-63b628e4d1a0 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:18.665623 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a5e8fa-dcd1-4ef7-ba78-63b628e4d1a0 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:18.665918 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a5e8fa-dcd1-4ef7-ba78-63b628e4d1a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:18.693141 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd010d7-2918-4981-bea1-62f6e4d8fa25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:18.694556 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd010d7-2918-4981-bea1-62f6e4d8fa25 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:18.694840 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd010d7-2918-4981-bea1-62f6e4d8fa25 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:18.695124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd010d7-2918-4981-bea1-62f6e4d8fa25 admin admin] PATH_INFO: `/v3/projects/0d647ac8f50049dab664e81a70e106ca` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:18.696942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd010d7-2918-4981-bea1-62f6e4d8fa25 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0d647ac8f50049dab664e81a70e106ca)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:18.699315 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd010d7-2918-4981-bea1-62f6e4d8fa25 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:18.702167 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-a0a5e8fa-dcd1-4ef7-ba78-63b628e4d1a0 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:18.704182 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a0a5e8fa-dcd1-4ef7-ba78-63b628e4d1a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:18.727168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 964/1960] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:59:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:59:18.735916 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e59ff0-99a6-418b-913d-6aef77bff6b7 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:18.736588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e59ff0-99a6-418b-913d-6aef77bff6b7 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:18.737146 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e59ff0-99a6-418b-913d-6aef77bff6b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:18.748001 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:59:18.764704 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-68e59ff0-99a6-418b-913d-6aef77bff6b7 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:18.766184 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-68e59ff0-99a6-418b-913d-6aef77bff6b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:18.776675 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-9fd010d7-2918-4981-bea1-62f6e4d8fa25 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0d647ac8f50049dab664e81a70e106ca', 'request_id': 'req-9fd010d7-2918-4981-bea1-62f6e4d8fa25'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:18.785757 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-9fd010d7-2918-4981-bea1-62f6e4d8fa25 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0d647ac8f50049dab664e81a70e106ca', 'request_id': 'req-9fd010d7-2918-4981-bea1-62f6e4d8fa25'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:18.792682 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-9fd010d7-2918-4981-bea1-62f6e4d8fa25 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0d647ac8f50049dab664e81a70e106ca', 'request_id': 'req-9fd010d7-2918-4981-bea1-62f6e4d8fa25'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:18.809230 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-9fd010d7-2918-4981-bea1-62f6e4d8fa25 admin admin] The token cache is being invalidate because project 0d647ac8f50049dab664e81a70e106ca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:59:18.810543 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-9fd010d7-2918-4981-bea1-62f6e4d8fa25 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:18.811702 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-9fd010d7-2918-4981-bea1-62f6e4d8fa25 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:18.813865 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 997/1961] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:59:18 2022] DELETE /identity/v3/projects/0d647ac8f50049dab664e81a70e106ca => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:59:18.847120 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 965/1962] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:59:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:59:18.851940 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd010d7-2918-4981-bea1-62f6e4d8fa25 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:18.868777 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd010d7-2918-4981-bea1-62f6e4d8fa25 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:18.875009 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b500dfc5-d8a0-4450-baa0-b299861d749d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:18.876046 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b500dfc5-d8a0-4450-baa0-b299861d749d admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:18.876552 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b500dfc5-d8a0-4450-baa0-b299861d749d admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:18.877865 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b500dfc5-d8a0-4450-baa0-b299861d749d admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:18.879803 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b500dfc5-d8a0-4450-baa0-b299861d749d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 14:59:18.880521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b500dfc5-d8a0-4450-baa0-b299861d749d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:18.882510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b500dfc5-d8a0-4450-baa0-b299861d749d admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:18.911019 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 998/1963] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 14:59:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 14:59:18.911488 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e59ff0-99a6-418b-913d-6aef77bff6b7 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:18.924347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e59ff0-99a6-418b-913d-6aef77bff6b7 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:18.930267 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca4a31f-f6fd-42ae-b536-75d27e014852 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:18.931117 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca4a31f-f6fd-42ae-b536-75d27e014852 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:18.933083 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca4a31f-f6fd-42ae-b536-75d27e014852 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:18.933684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca4a31f-f6fd-42ae-b536-75d27e014852 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:18.934698 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca4a31f-f6fd-42ae-b536-75d27e014852 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:18.938848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca4a31f-f6fd-42ae-b536-75d27e014852 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:18.982701 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 966/1964] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 14:59:18 2022] POST /identity/v3/projects => generated 373 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 14:59:18.988162 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b500dfc5-d8a0-4450-baa0-b299861d749d admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:19.015136 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b500dfc5-d8a0-4450-baa0-b299861d749d admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:19.106626 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79572560-f76d-42c9-b5c7-e5ad12943cd2 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:19.106626 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79572560-f76d-42c9-b5c7-e5ad12943cd2 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:19.106626 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79572560-f76d-42c9-b5c7-e5ad12943cd2 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:19.108100 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 967/1965] 10.222.0.36 () {54 vars in 921 bytes} [Mon Oct 31 14:59:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 14:59:19.127947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102759f6-1d0f-430c-8ed7-41ed8aa2f89b None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:19.128233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102759f6-1d0f-430c-8ed7-41ed8aa2f89b None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:19.128514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102759f6-1d0f-430c-8ed7-41ed8aa2f89b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:19.140554 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961f9f5a-7529-41d9-9945-5e3cf844873c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:19.141246 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f9f5a-7529-41d9-9945-5e3cf844873c admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:19.141530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f9f5a-7529-41d9-9945-5e3cf844873c admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:19.141851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f9f5a-7529-41d9-9945-5e3cf844873c admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:19.142503 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961f9f5a-7529-41d9-9945-5e3cf844873c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:19.152597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961f9f5a-7529-41d9-9945-5e3cf844873c admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:19.202904 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 999/1966] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 14:59:18 2022] POST /identity/v3/users => generated 287 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 14:59:19.207438 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-102759f6-1d0f-430c-8ed7-41ed8aa2f89b None None] MFA Rules not processed for user `c8cace029a92406f850bb0a4ca07c461`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:19.208246 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961f9f5a-7529-41d9-9945-5e3cf844873c admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:19.210041 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-102759f6-1d0f-430c-8ed7-41ed8aa2f89b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:19.224527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961f9f5a-7529-41d9-9945-5e3cf844873c admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:19.240615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23fca18-07ef-495c-a4df-b35421657cb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:19.241251 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23fca18-07ef-495c-a4df-b35421657cb2 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:19.241529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23fca18-07ef-495c-a4df-b35421657cb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:19.241801 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23fca18-07ef-495c-a4df-b35421657cb2 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:19.242506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23fca18-07ef-495c-a4df-b35421657cb2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:19.244265 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23fca18-07ef-495c-a4df-b35421657cb2 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:19.255439 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1000/1967] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:59:19 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 14:59:19.263527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23fca18-07ef-495c-a4df-b35421657cb2 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:19.282133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:59:19.283251 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23fca18-07ef-495c-a4df-b35421657cb2 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:19.299177 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04dd1b5-7183-42a4-8a99-592c9e46d08e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:19.307839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04dd1b5-7183-42a4-8a99-592c9e46d08e admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:19.309627 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04dd1b5-7183-42a4-8a99-592c9e46d08e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:19.310015 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04dd1b5-7183-42a4-8a99-592c9e46d08e admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:19.310788 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04dd1b5-7183-42a4-8a99-592c9e46d08e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:19.312642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04dd1b5-7183-42a4-8a99-592c9e46d08e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:19.325137 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 968/1968] 10.222.0.36 () {58 vars in 1018 bytes} [Mon Oct 31 14:59:19 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:59:19.337846 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102759f6-1d0f-430c-8ed7-41ed8aa2f89b None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:19.341500 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-102759f6-1d0f-430c-8ed7-41ed8aa2f89b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:19.344783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1001/1969] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 14:59:19 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 14:59:19.345748 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04dd1b5-7183-42a4-8a99-592c9e46d08e admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:19.359651 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04dd1b5-7183-42a4-8a99-592c9e46d08e admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:19.372307 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03198a53-ab1a-4dd0-b408-e16848f0cf9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:19.373236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03198a53-ab1a-4dd0-b408-e16848f0cf9d admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:19.376007 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03198a53-ab1a-4dd0-b408-e16848f0cf9d admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:19.376493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03198a53-ab1a-4dd0-b408-e16848f0cf9d admin admin] PATH_INFO: `/v3/projects/6157f17410284e6eae8b53366e876352/users/7c229269abd34c1ead739a11cce445b8/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:19.403092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03198a53-ab1a-4dd0-b408-e16848f0cf9d admin admin] RBAC: Authorizing `identity:create_grant(project_id=6157f17410284e6eae8b53366e876352, user_id=7c229269abd34c1ead739a11cce445b8, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:19.406550 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03198a53-ab1a-4dd0-b408-e16848f0cf9d admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:19.428087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1002/1970] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 14:59:19 2022] PUT /identity/v3/projects/6157f17410284e6eae8b53366e876352/users/7c229269abd34c1ead739a11cce445b8/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:59:19.449659 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8689df-bc2c-41b4-93cc-ed8617e1cd5b None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:19.453781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8689df-bc2c-41b4-93cc-ed8617e1cd5b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:19.454866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8689df-bc2c-41b4-93cc-ed8617e1cd5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:19.509665 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102759f6-1d0f-430c-8ed7-41ed8aa2f89b None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:19.527597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-cb8689df-bc2c-41b4-93cc-ed8617e1cd5b None None] MFA Rules not processed for user `7c229269abd34c1ead739a11cce445b8`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:19.532984 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-cb8689df-bc2c-41b4-93cc-ed8617e1cd5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:19.609505 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:59:19.635245 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a8ace3-babe-4d95-9ec7-bb9e5ce80ee9 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9808dcba0b64027bb70e31d1569957d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:19.636535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a8ace3-babe-4d95-9ec7-bb9e5ce80ee9 service glance] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:19.636980 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a8ace3-babe-4d95-9ec7-bb9e5ce80ee9 service glance] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:19.637421 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a8ace3-babe-4d95-9ec7-bb9e5ce80ee9 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:19.647493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1003/1971] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:59:19 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:59:19.661274 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8689df-bc2c-41b4-93cc-ed8617e1cd5b None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:19.663021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-cb8689df-bc2c-41b4-93cc-ed8617e1cd5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:19.738274 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a8ace3-babe-4d95-9ec7-bb9e5ce80ee9 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:19.740448 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a8ace3-babe-4d95-9ec7-bb9e5ce80ee9 service glance] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:19.763982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 969/1972] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:59:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 14:59:19.775219 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8689df-bc2c-41b4-93cc-ed8617e1cd5b None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:19.781386 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b376c4-4792-422d-b170-8535f9b7eeda service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:19.782185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b376c4-4792-422d-b170-8535f9b7eeda service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:19.782503 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b376c4-4792-422d-b170-8535f9b7eeda service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:19.782781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b376c4-4792-422d-b170-8535f9b7eeda service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:19.793213 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b376c4-4792-422d-b170-8535f9b7eeda service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:19.794663 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b376c4-4792-422d-b170-8535f9b7eeda service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:19.815804 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1004/1973] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:59:19 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:59:19.843830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199beb54-26c9-4dbe-803b-976e1eed057f None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:19.843830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199beb54-26c9-4dbe-803b-976e1eed057f None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:19.843830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199beb54-26c9-4dbe-803b-976e1eed057f None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:19.843830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 970/1974] 10.222.0.36 () {54 vars in 921 bytes} [Mon Oct 31 14:59:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:59:19.848066 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04873dbc-73d5-4f9b-b618-8f7734cc814f None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:19.848570 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04873dbc-73d5-4f9b-b618-8f7734cc814f None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:19.848881 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04873dbc-73d5-4f9b-b618-8f7734cc814f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:19.869218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-04873dbc-73d5-4f9b-b618-8f7734cc814f None None] MFA Rules not processed for user `14946d97fd384e2a8bff68c4f53c2cfe`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:19.870453 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-04873dbc-73d5-4f9b-b618-8f7734cc814f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:19.931939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1005/1975] 10.222.0.36 () {58 vars in 1018 bytes} [Mon Oct 31 14:59:19 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:59:19.944622 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199beb54-26c9-4dbe-803b-976e1eed057f None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:19.946320 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-199beb54-26c9-4dbe-803b-976e1eed057f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:19.962034 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199beb54-26c9-4dbe-803b-976e1eed057f None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:19.968432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9e3f80-0aeb-4c3f-81aa-bd1c68dd2eea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:19.969192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9e3f80-0aeb-4c3f-81aa-bd1c68dd2eea service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:19.969483 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9e3f80-0aeb-4c3f-81aa-bd1c68dd2eea service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:19.969755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9e3f80-0aeb-4c3f-81aa-bd1c68dd2eea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:19.978998 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9e3f80-0aeb-4c3f-81aa-bd1c68dd2eea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:19.980726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9e3f80-0aeb-4c3f-81aa-bd1c68dd2eea service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:19.984061 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:59:19.991581 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 971/1976] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:59:19 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:59:20.120816 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf6974d-24b4-4301-b5f9-e915bf346ef8 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:20.121281 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf6974d-24b4-4301-b5f9-e915bf346ef8 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:20.121651 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf6974d-24b4-4301-b5f9-e915bf346ef8 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:20.125201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1006/1977] 10.222.0.36 () {54 vars in 921 bytes} [Mon Oct 31 14:59:20 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 14:59:20.131044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d40c205-726a-4f5c-8245-794607bc2e53 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:20.131329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d40c205-726a-4f5c-8245-794607bc2e53 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:20.131644 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d40c205-726a-4f5c-8245-794607bc2e53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:20.156070 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-9d40c205-726a-4f5c-8245-794607bc2e53 None None] MFA Rules not processed for user `c8cace029a92406f850bb0a4ca07c461`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:20.157299 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-9d40c205-726a-4f5c-8245-794607bc2e53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:20.170406 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 972/1978] 10.222.0.36 () {58 vars in 1018 bytes} [Mon Oct 31 14:59:20 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 14:59:20.183226 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf6974d-24b4-4301-b5f9-e915bf346ef8 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:20.202650 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf6974d-24b4-4301-b5f9-e915bf346ef8 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:20.214550 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd13329-527c-4f49-a1f5-77eae50bcecd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:20.215473 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd13329-527c-4f49-a1f5-77eae50bcecd service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:20.216793 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd13329-527c-4f49-a1f5-77eae50bcecd service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:20.217397 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd13329-527c-4f49-a1f5-77eae50bcecd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:20.246652 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd13329-527c-4f49-a1f5-77eae50bcecd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:20.248591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd13329-527c-4f49-a1f5-77eae50bcecd service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:20.275474 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1007/1979] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:59:20 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:59:25.712248 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade14b74-58d1-4522-888e-4fb4a820af48 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:25.713180 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade14b74-58d1-4522-888e-4fb4a820af48 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:25.713843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade14b74-58d1-4522-888e-4fb4a820af48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:25.737028 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-ade14b74-58d1-4522-888e-4fb4a820af48 None None] MFA Rules not processed for user `e0d0f8598fac40449bcb2a55ff8df8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:25.738159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ade14b74-58d1-4522-888e-4fb4a820af48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:25.823953 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 973/1980] 10.222.0.36 () {58 vars in 1009 bytes} [Mon Oct 31 14:59:25 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:59:25.842364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd13329-527c-4f49-a1f5-77eae50bcecd service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:25.843906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-6dd13329-527c-4f49-a1f5-77eae50bcecd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:25.942936 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd13329-527c-4f49-a1f5-77eae50bcecd service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:25.948124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61bbbd7-8872-4f42-92bc-45a7413afd53 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4ae5e20cc5b40ccbdb951f908f5080d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:25.948736 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61bbbd7-8872-4f42-92bc-45a7413afd53 service swift] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:25.949017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61bbbd7-8872-4f42-92bc-45a7413afd53 service swift] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:25.949294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61bbbd7-8872-4f42-92bc-45a7413afd53 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:25.960249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61bbbd7-8872-4f42-92bc-45a7413afd53 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:25.962216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61bbbd7-8872-4f42-92bc-45a7413afd53 service swift] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:25.972742 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1008/1981] 10.222.0.36 () {60 vars in 1384 bytes} [Mon Oct 31 14:59:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 31 14:59:30.979117 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade14b74-58d1-4522-888e-4fb4a820af48 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:31.195141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade14b74-58d1-4522-888e-4fb4a820af48 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:31.210228 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7de6cd-281c-4296-bc4c-0e143e2ff449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:31.210904 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7de6cd-281c-4296-bc4c-0e143e2ff449 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:31.211192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7de6cd-281c-4296-bc4c-0e143e2ff449 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:31.211501 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7de6cd-281c-4296-bc4c-0e143e2ff449 admin admin] PATH_INFO: `/v3/users/7c229269abd34c1ead739a11cce445b8` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:31.212709 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7de6cd-281c-4296-bc4c-0e143e2ff449 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7c229269abd34c1ead739a11cce445b8)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:31.216042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7de6cd-281c-4296-bc4c-0e143e2ff449 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:31.310820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7b7de6cd-281c-4296-bc4c-0e143e2ff449 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7c229269abd34c1ead739a11cce445b8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:31.332670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7b7de6cd-281c-4296-bc4c-0e143e2ff449 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7c229269abd34c1ead739a11cce445b8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:31.333958 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7b7de6cd-281c-4296-bc4c-0e143e2ff449 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7c229269abd34c1ead739a11cce445b8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:31.335439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7b7de6cd-281c-4296-bc4c-0e143e2ff449 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7c229269abd34c1ead739a11cce445b8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:31.367458 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7b7de6cd-281c-4296-bc4c-0e143e2ff449 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7c229269abd34c1ead739a11cce445b8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:31.388299 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 974/1982] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 14:59:30 2022] DELETE /identity/v3/users/7c229269abd34c1ead739a11cce445b8 => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:59:31.392480 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b8bbf2-89e6-4f55-8711-82bf22a456c3 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:31.392911 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b8bbf2-89e6-4f55-8711-82bf22a456c3 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:31.393009 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b8bbf2-89e6-4f55-8711-82bf22a456c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:31.433159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-24b8bbf2-89e6-4f55-8711-82bf22a456c3 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:31.434480 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-24b8bbf2-89e6-4f55-8711-82bf22a456c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:31.438673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:59:31.525771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1009/1983] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 14:59:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:59:31.537956 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7de6cd-281c-4296-bc4c-0e143e2ff449 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:31.537956 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-7b7de6cd-281c-4296-bc4c-0e143e2ff449 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:31.563961 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:59:31.613581 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7de6cd-281c-4296-bc4c-0e143e2ff449 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:31.617148 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015e9951-7d0c-4eec-9cae-6a760ca38867 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:31.617761 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015e9951-7d0c-4eec-9cae-6a760ca38867 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:31.618067 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015e9951-7d0c-4eec-9cae-6a760ca38867 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:31.618318 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015e9951-7d0c-4eec-9cae-6a760ca38867 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:31.629105 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015e9951-7d0c-4eec-9cae-6a760ca38867 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:31.630830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015e9951-7d0c-4eec-9cae-6a760ca38867 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:31.644225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 975/1984] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:59:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:59:31.941038 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b8bbf2-89e6-4f55-8711-82bf22a456c3 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:31.942722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-24b8bbf2-89e6-4f55-8711-82bf22a456c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:31.961605 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b8bbf2-89e6-4f55-8711-82bf22a456c3 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:31.980364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac38050a-a6d5-4ecc-a3f1-1fa5bde2fb4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:31.981040 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac38050a-a6d5-4ecc-a3f1-1fa5bde2fb4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:31.981327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac38050a-a6d5-4ecc-a3f1-1fa5bde2fb4a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:31.981606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac38050a-a6d5-4ecc-a3f1-1fa5bde2fb4a admin admin] PATH_INFO: `/v3/projects/6157f17410284e6eae8b53366e876352` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:31.982839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac38050a-a6d5-4ecc-a3f1-1fa5bde2fb4a admin admin] RBAC: Authorizing `identity:delete_project(project_id=6157f17410284e6eae8b53366e876352)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:31.984967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac38050a-a6d5-4ecc-a3f1-1fa5bde2fb4a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:32.042059 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ac38050a-a6d5-4ecc-a3f1-1fa5bde2fb4a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6157f17410284e6eae8b53366e876352', 'request_id': 'req-ac38050a-a6d5-4ecc-a3f1-1fa5bde2fb4a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:32.053536 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ac38050a-a6d5-4ecc-a3f1-1fa5bde2fb4a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6157f17410284e6eae8b53366e876352', 'request_id': 'req-ac38050a-a6d5-4ecc-a3f1-1fa5bde2fb4a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:32.061951 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ac38050a-a6d5-4ecc-a3f1-1fa5bde2fb4a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6157f17410284e6eae8b53366e876352', 'request_id': 'req-ac38050a-a6d5-4ecc-a3f1-1fa5bde2fb4a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:32.090441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ac38050a-a6d5-4ecc-a3f1-1fa5bde2fb4a admin admin] The token cache is being invalidate because project 6157f17410284e6eae8b53366e876352 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 14:59:32.090738 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ac38050a-a6d5-4ecc-a3f1-1fa5bde2fb4a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:32.091362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ac38050a-a6d5-4ecc-a3f1-1fa5bde2fb4a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 14:59:32.092877 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1010/1985] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 14:59:31 2022] DELETE /identity/v3/projects/6157f17410284e6eae8b53366e876352 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 14:59:40.668481 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015e9951-7d0c-4eec-9cae-6a760ca38867 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:40.671584 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-015e9951-7d0c-4eec-9cae-6a760ca38867 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:40.772494 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015e9951-7d0c-4eec-9cae-6a760ca38867 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:40.845967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b539fa43-645c-40ac-a62a-b4d31ddd8e38 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70ea2aaf529b46dabb70b71f4dcc3860', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:40.847103 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b539fa43-645c-40ac-a62a-b4d31ddd8e38 service placement] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:40.847584 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b539fa43-645c-40ac-a62a-b4d31ddd8e38 service placement] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:40.847715 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b539fa43-645c-40ac-a62a-b4d31ddd8e38 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:40.851048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-b539fa43-645c-40ac-a62a-b4d31ddd8e38 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:40.865907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b539fa43-645c-40ac-a62a-b4d31ddd8e38 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:40.867177 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b539fa43-645c-40ac-a62a-b4d31ddd8e38 service placement] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:40.879921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 976/1986] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 14:59:40 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 14:59:42.087231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=50,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:59:46.641519 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d387a6ae-4344-44b1-8c20-a9936f47c586 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:46.642060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d387a6ae-4344-44b1-8c20-a9936f47c586 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:46.642060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d387a6ae-4344-44b1-8c20-a9936f47c586 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:46.662013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-d387a6ae-4344-44b1-8c20-a9936f47c586 None None] MFA Rules not processed for user `e0d0f8598fac40449bcb2a55ff8df8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 14:59:46.663242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d387a6ae-4344-44b1-8c20-a9936f47c586 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:46.719018 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1011/1987] 10.222.0.36 () {58 vars in 967 bytes} [Mon Oct 31 14:59:46 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 14:59:46.731880 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b539fa43-645c-40ac-a62a-b4d31ddd8e38 service placement] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 14:59:46.733242 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-b539fa43-645c-40ac-a62a-b4d31ddd8e38 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 14:59:46.783309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:59:46.800619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b539fa43-645c-40ac-a62a-b4d31ddd8e38 service placement] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 14:59:46.805493 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af1b924-17c5-4349-a209-bd8ab2fd401a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4ae5e20cc5b40ccbdb951f908f5080d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 14:59:46.806095 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af1b924-17c5-4349-a209-bd8ab2fd401a service swift] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 14:59:46.806372 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af1b924-17c5-4349-a209-bd8ab2fd401a service swift] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 14:59:46.806666 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af1b924-17c5-4349-a209-bd8ab2fd401a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 14:59:46.816174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af1b924-17c5-4349-a209-bd8ab2fd401a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 14:59:46.817770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af1b924-17c5-4349-a209-bd8ab2fd401a service swift] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 14:59:46.825951 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 977/1988] 10.222.0.36 () {60 vars in 1384 bytes} [Mon Oct 31 14:59:46 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 31 14:59:56.712114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=25 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 14:59:56.822700 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=9 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:00:40.717038 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c362737-91d5-4db2-94d9-b8043e394092 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:00:40.717524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c362737-91d5-4db2-94d9-b8043e394092 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:00:40.717607 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c362737-91d5-4db2-94d9-b8043e394092 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:00:40.739726 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-0c362737-91d5-4db2-94d9-b8043e394092 None None] MFA Rules not processed for user `cdcbcb2ed9e44dbe8444e531fc1eb707`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:00:40.741327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0c362737-91d5-4db2-94d9-b8043e394092 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:00:40.799429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1012/1989] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:00:40 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:00:41.294245 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e045e8-e55d-4f36-8628-aa6c3c8342ec None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:00:41.295543 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e045e8-e55d-4f36-8628-aa6c3c8342ec None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:00:41.296030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e045e8-e55d-4f36-8628-aa6c3c8342ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:00:41.318517 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-d3e045e8-e55d-4f36-8628-aa6c3c8342ec None None] MFA Rules not processed for user `e1fb0b1657cd43baaa93a16ce6d9ec8f`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:00:41.322966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d3e045e8-e55d-4f36-8628-aa6c3c8342ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:00:41.379194 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 978/1990] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:00:41 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:00:50.791058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=25 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:00:51.372013 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=25 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:00:56.171744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967eeb58-8c85-4f6a-a133-cbf870f5185c None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:00:56.172593 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967eeb58-8c85-4f6a-a133-cbf870f5185c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:00:56.173034 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967eeb58-8c85-4f6a-a133-cbf870f5185c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:00:56.192370 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-967eeb58-8c85-4f6a-a133-cbf870f5185c None None] MFA Rules not processed for user `e6bfe2c1ba054935ae8e9fcc054faf68`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:00:56.193598 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-967eeb58-8c85-4f6a-a133-cbf870f5185c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:00:56.257182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1013/1991] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:00:56 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:00:58.687319 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c2afa0-1dd9-4b5d-bff9-6e35566ab74f None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:00:58.687851 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c2afa0-1dd9-4b5d-bff9-6e35566ab74f None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:00:58.688233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c2afa0-1dd9-4b5d-bff9-6e35566ab74f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:00:58.709122 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-40c2afa0-1dd9-4b5d-bff9-6e35566ab74f None None] MFA Rules not processed for user `75d1901be8064ff5bf79f0268d589e10`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:00:58.710652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-40c2afa0-1dd9-4b5d-bff9-6e35566ab74f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:00:58.775418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 979/1992] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:00:58 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:01:06.248961 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=25 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:01:08.768684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=25 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:01:40.347204 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967eeb58-8c85-4f6a-a133-cbf870f5185c None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:01:40.348635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-967eeb58-8c85-4f6a-a133-cbf870f5185c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:01:40.404806 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967eeb58-8c85-4f6a-a133-cbf870f5185c None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:01:40.408385 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83300803-bbe3-4fc0-abdc-595357a916e1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:01:40.408987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83300803-bbe3-4fc0-abdc-595357a916e1 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:01:40.409267 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83300803-bbe3-4fc0-abdc-595357a916e1 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:01:40.409541 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83300803-bbe3-4fc0-abdc-595357a916e1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:01:40.411257 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-83300803-bbe3-4fc0-abdc-595357a916e1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:01:40.481636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83300803-bbe3-4fc0-abdc-595357a916e1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:01:40.483107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83300803-bbe3-4fc0-abdc-595357a916e1 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:01:40.494186 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1014/1993] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:01:40 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:01:41.257210 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c2afa0-1dd9-4b5d-bff9-6e35566ab74f None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:01:41.258733 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-40c2afa0-1dd9-4b5d-bff9-6e35566ab74f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:01:41.322091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c2afa0-1dd9-4b5d-bff9-6e35566ab74f None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:01:41.326049 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490d8935-9baf-4830-8cc8-afaa87170e8d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:01:41.326646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490d8935-9baf-4830-8cc8-afaa87170e8d service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:01:41.326925 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490d8935-9baf-4830-8cc8-afaa87170e8d service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:01:41.327198 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490d8935-9baf-4830-8cc8-afaa87170e8d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:01:41.328958 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-490d8935-9baf-4830-8cc8-afaa87170e8d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:01:41.342646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490d8935-9baf-4830-8cc8-afaa87170e8d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:01:41.344104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490d8935-9baf-4830-8cc8-afaa87170e8d service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:01:41.354809 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 980/1994] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:01:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:01:42.803768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83300803-bbe3-4fc0-abdc-595357a916e1 service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:01:42.805215 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-83300803-bbe3-4fc0-abdc-595357a916e1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:01:42.820429 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83300803-bbe3-4fc0-abdc-595357a916e1 service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:01:42.826234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b8db47-499e-4d85-b57e-a93ee6f3987a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:01:42.826826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b8db47-499e-4d85-b57e-a93ee6f3987a service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:01:42.827105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b8db47-499e-4d85-b57e-a93ee6f3987a service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:01:42.827400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b8db47-499e-4d85-b57e-a93ee6f3987a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:01:42.829141 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e1b8db47-499e-4d85-b57e-a93ee6f3987a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:01:42.839353 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b8db47-499e-4d85-b57e-a93ee6f3987a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:01:42.841529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b8db47-499e-4d85-b57e-a93ee6f3987a service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:01:42.850317 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1015/1995] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:01:42 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:01:51.350053 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=26 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:01:52.846134 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=60 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:01:53.694161 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490d8935-9baf-4830-8cc8-afaa87170e8d service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:01:53.704944 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490d8935-9baf-4830-8cc8-afaa87170e8d service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:01:53.709352 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e053d16-b317-4a2c-9cbf-7d3e831152de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:01:53.709953 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e053d16-b317-4a2c-9cbf-7d3e831152de service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:01:53.710786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e053d16-b317-4a2c-9cbf-7d3e831152de service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:01:53.711174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e053d16-b317-4a2c-9cbf-7d3e831152de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:01:53.713420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-9e053d16-b317-4a2c-9cbf-7d3e831152de service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:01:53.725901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e053d16-b317-4a2c-9cbf-7d3e831152de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:01:53.727335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e053d16-b317-4a2c-9cbf-7d3e831152de service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:01:53.741793 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 981/1996] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:01:53 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:01:53.743283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b8db47-499e-4d85-b57e-a93ee6f3987a service nova] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:01:53.753899 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b8db47-499e-4d85-b57e-a93ee6f3987a service nova] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:01:53.757477 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9290a644-9ccf-4b2d-99fc-c692b3ed25fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:01:53.758074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9290a644-9ccf-4b2d-99fc-c692b3ed25fd service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:01:53.758355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9290a644-9ccf-4b2d-99fc-c692b3ed25fd service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:01:53.758626 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9290a644-9ccf-4b2d-99fc-c692b3ed25fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:01:53.760427 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-9290a644-9ccf-4b2d-99fc-c692b3ed25fd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:01:53.772231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9290a644-9ccf-4b2d-99fc-c692b3ed25fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:01:53.773642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9290a644-9ccf-4b2d-99fc-c692b3ed25fd service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:01:53.783110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1016/1997] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:01:53 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:01:54.843335 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e053d16-b317-4a2c-9cbf-7d3e831152de service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:01:54.844762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-9e053d16-b317-4a2c-9cbf-7d3e831152de service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:01:54.860679 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e053d16-b317-4a2c-9cbf-7d3e831152de service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:01:54.866463 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235d8a11-eb95-4b4b-b98e-f4c0c8ebaee5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:01:54.867145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235d8a11-eb95-4b4b-b98e-f4c0c8ebaee5 service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:01:54.867435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235d8a11-eb95-4b4b-b98e-f4c0c8ebaee5 service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:01:54.867747 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235d8a11-eb95-4b4b-b98e-f4c0c8ebaee5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:01:54.870981 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-235d8a11-eb95-4b4b-b98e-f4c0c8ebaee5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:01:54.882119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235d8a11-eb95-4b4b-b98e-f4c0c8ebaee5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:01:54.883757 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235d8a11-eb95-4b4b-b98e-f4c0c8ebaee5 service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:01:54.894139 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 982/1998] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:01:54 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:01.764890 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9290a644-9ccf-4b2d-99fc-c692b3ed25fd service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:01.767921 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-9290a644-9ccf-4b2d-99fc-c692b3ed25fd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:01.789570 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9290a644-9ccf-4b2d-99fc-c692b3ed25fd service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:01.802417 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3423ea18-2054-4b8f-a292-c0409f874533 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:01.804840 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3423ea18-2054-4b8f-a292-c0409f874533 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:01.805209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3423ea18-2054-4b8f-a292-c0409f874533 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:01.805562 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3423ea18-2054-4b8f-a292-c0409f874533 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:01.811268 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-3423ea18-2054-4b8f-a292-c0409f874533 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:01.913341 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3423ea18-2054-4b8f-a292-c0409f874533 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:01.914863 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3423ea18-2054-4b8f-a292-c0409f874533 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:01.926049 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1017/1999] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:02.351765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4355e8-8503-48af-9821-229cab4024ad None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:02.355779 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4355e8-8503-48af-9821-229cab4024ad None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:02.355779 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4355e8-8503-48af-9821-229cab4024ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:02.379017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-dd4355e8-8503-48af-9821-229cab4024ad None None] MFA Rules not processed for user `e0d0f8598fac40449bcb2a55ff8df8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:02.380447 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-dd4355e8-8503-48af-9821-229cab4024ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:02.388630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 983/2000] 10.222.0.36 () {58 vars in 1009 bytes} [Mon Oct 31 15:02:02 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:02.443748 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3423ea18-2054-4b8f-a292-c0409f874533 service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:02.456494 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3423ea18-2054-4b8f-a292-c0409f874533 service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:02.461853 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b6995c-812e-4f09-9941-ac1a7452313f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4ae5e20cc5b40ccbdb951f908f5080d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:02.462492 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b6995c-812e-4f09-9941-ac1a7452313f service swift] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:02.462778 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b6995c-812e-4f09-9941-ac1a7452313f service swift] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:02.463057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b6995c-812e-4f09-9941-ac1a7452313f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:02.473915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b6995c-812e-4f09-9941-ac1a7452313f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:02.475659 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b6995c-812e-4f09-9941-ac1a7452313f service swift] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:02.485668 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1018/2001] 10.222.0.36 () {60 vars in 1384 bytes} [Mon Oct 31 15:02:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 31 15:02:06.073309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3129a8d-d0b5-4364-acbd-762607a46f0f None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:06.073804 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3129a8d-d0b5-4364-acbd-762607a46f0f None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:06.074024 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3129a8d-d0b5-4364-acbd-762607a46f0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:06.126287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-c3129a8d-d0b5-4364-acbd-762607a46f0f None None] MFA Rules not processed for user `cdcbcb2ed9e44dbe8444e531fc1eb707`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:06.127752 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-c3129a8d-d0b5-4364-acbd-762607a46f0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:06.164667 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 984/2002] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:06 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:02:06.859179 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b6995c-812e-4f09-9941-ac1a7452313f service swift] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:06.871086 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b6995c-812e-4f09-9941-ac1a7452313f service swift] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:06.880809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1efdff3-3883-46d5-9bb4-340881d1d050 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:06.881575 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1efdff3-3883-46d5-9bb4-340881d1d050 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:06.881959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1efdff3-3883-46d5-9bb4-340881d1d050 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:06.882332 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1efdff3-3883-46d5-9bb4-340881d1d050 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:06.887592 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d1efdff3-3883-46d5-9bb4-340881d1d050 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:06.909567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1efdff3-3883-46d5-9bb4-340881d1d050 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:06.911245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1efdff3-3883-46d5-9bb4-340881d1d050 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:06.934494 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1019/2003] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:06 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 76 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:07.223313 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3129a8d-d0b5-4364-acbd-762607a46f0f None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:07.224870 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-c3129a8d-d0b5-4364-acbd-762607a46f0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:07.249085 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3129a8d-d0b5-4364-acbd-762607a46f0f None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:07.254969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d40578-53fe-4070-98f8-32d9cd3a8a15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:07.255452 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d40578-53fe-4070-98f8-32d9cd3a8a15 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:07.256018 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d40578-53fe-4070-98f8-32d9cd3a8a15 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:07.256564 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d40578-53fe-4070-98f8-32d9cd3a8a15 admin admin] PATH_INFO: `/v3/users/e6bfe2c1ba054935ae8e9fcc054faf68` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:07.259320 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d40578-53fe-4070-98f8-32d9cd3a8a15 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e6bfe2c1ba054935ae8e9fcc054faf68)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:07.263001 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d40578-53fe-4070-98f8-32d9cd3a8a15 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:07.446693 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-96d40578-53fe-4070-98f8-32d9cd3a8a15 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e6bfe2c1ba054935ae8e9fcc054faf68'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:07.448200 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1efdff3-3883-46d5-9bb4-340881d1d050 service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:07.449661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d1efdff3-3883-46d5-9bb4-340881d1d050 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:07.476286 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-96d40578-53fe-4070-98f8-32d9cd3a8a15 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e6bfe2c1ba054935ae8e9fcc054faf68'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:07.477071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-96d40578-53fe-4070-98f8-32d9cd3a8a15 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e6bfe2c1ba054935ae8e9fcc054faf68'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:07.477701 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-96d40578-53fe-4070-98f8-32d9cd3a8a15 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e6bfe2c1ba054935ae8e9fcc054faf68'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:07.484696 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1efdff3-3883-46d5-9bb4-340881d1d050 service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:07.492295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-96d40578-53fe-4070-98f8-32d9cd3a8a15 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e6bfe2c1ba054935ae8e9fcc054faf68'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:07.492442 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdf9401-12b8-477e-b058-f69163357dd4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70ea2aaf529b46dabb70b71f4dcc3860', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:07.492994 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdf9401-12b8-477e-b058-f69163357dd4 service placement] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:07.493277 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdf9401-12b8-477e-b058-f69163357dd4 service placement] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:07.493551 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdf9401-12b8-477e-b058-f69163357dd4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:07.495848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-fcdf9401-12b8-477e-b058-f69163357dd4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:07.550452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcdf9401-12b8-477e-b058-f69163357dd4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:07.551882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 985/2004] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:07 2022] DELETE /identity/v3/users/e6bfe2c1ba054935ae8e9fcc054faf68 => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:07.553025 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcdf9401-12b8-477e-b058-f69163357dd4 service placement] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:07.578879 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1020/2005] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:07 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:07.919295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d40578-53fe-4070-98f8-32d9cd3a8a15 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:07.921397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-96d40578-53fe-4070-98f8-32d9cd3a8a15 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:08.051207 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d40578-53fe-4070-98f8-32d9cd3a8a15 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:08.064482 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a29c57a-1325-4807-91d6-f4f73bc45f5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:08.065655 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a29c57a-1325-4807-91d6-f4f73bc45f5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:08.066110 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a29c57a-1325-4807-91d6-f4f73bc45f5f admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:08.066548 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a29c57a-1325-4807-91d6-f4f73bc45f5f admin admin] PATH_INFO: `/v3/projects/8bc7c0eaebf146bea1af6b4d708bf62b` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:08.068060 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a29c57a-1325-4807-91d6-f4f73bc45f5f admin admin] RBAC: Authorizing `identity:delete_project(project_id=8bc7c0eaebf146bea1af6b4d708bf62b)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:08.070124 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a29c57a-1325-4807-91d6-f4f73bc45f5f admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:08.138223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:08.153466 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7a29c57a-1325-4807-91d6-f4f73bc45f5f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8bc7c0eaebf146bea1af6b4d708bf62b', 'request_id': 'req-7a29c57a-1325-4807-91d6-f4f73bc45f5f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:08.176621 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7a29c57a-1325-4807-91d6-f4f73bc45f5f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8bc7c0eaebf146bea1af6b4d708bf62b', 'request_id': 'req-7a29c57a-1325-4807-91d6-f4f73bc45f5f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:08.183593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7a29c57a-1325-4807-91d6-f4f73bc45f5f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8bc7c0eaebf146bea1af6b4d708bf62b', 'request_id': 'req-7a29c57a-1325-4807-91d6-f4f73bc45f5f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:08.197712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7a29c57a-1325-4807-91d6-f4f73bc45f5f admin admin] The token cache is being invalidate because project 8bc7c0eaebf146bea1af6b4d708bf62b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:08.198427 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7a29c57a-1325-4807-91d6-f4f73bc45f5f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:08.199453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7a29c57a-1325-4807-91d6-f4f73bc45f5f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:08.201893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 986/2006] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:07 2022] DELETE /identity/v3/projects/8bc7c0eaebf146bea1af6b4d708bf62b => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:08.208634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1448ffc4-87ea-4655-b674-cf9e12ea8e3e None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:08.209162 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1448ffc4-87ea-4655-b674-cf9e12ea8e3e None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:08.209162 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1448ffc4-87ea-4655-b674-cf9e12ea8e3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:08.235794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-1448ffc4-87ea-4655-b674-cf9e12ea8e3e None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:08.237018 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1448ffc4-87ea-4655-b674-cf9e12ea8e3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:08.318670 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1021/2007] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:08.330197 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e1b94f-3225-4721-aa8f-fec07efca4b2 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:08.330716 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e1b94f-3225-4721-aa8f-fec07efca4b2 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:08.330716 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e1b94f-3225-4721-aa8f-fec07efca4b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:08.362790 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-19e1b94f-3225-4721-aa8f-fec07efca4b2 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:08.370077 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-19e1b94f-3225-4721-aa8f-fec07efca4b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:08.399270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 987/2008] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:08.400922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1448ffc4-87ea-4655-b674-cf9e12ea8e3e None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:08.414587 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1448ffc4-87ea-4655-b674-cf9e12ea8e3e None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:08.421966 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb0af68-1287-46e8-938c-ee9023ae329e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:08.422838 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb0af68-1287-46e8-938c-ee9023ae329e admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:08.423122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb0af68-1287-46e8-938c-ee9023ae329e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:08.423399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb0af68-1287-46e8-938c-ee9023ae329e admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:08.424122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb0af68-1287-46e8-938c-ee9023ae329e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:02:08.424430 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb0af68-1287-46e8-938c-ee9023ae329e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:08.426035 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb0af68-1287-46e8-938c-ee9023ae329e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:08.454912 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1022/2009] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:02:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:02:08.458570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e1b94f-3225-4721-aa8f-fec07efca4b2 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:08.470007 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e1b94f-3225-4721-aa8f-fec07efca4b2 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:08.478060 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24e62a8-fbd0-478c-b1e9-be2c5a30dacf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:08.478959 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24e62a8-fbd0-478c-b1e9-be2c5a30dacf admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:08.479271 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24e62a8-fbd0-478c-b1e9-be2c5a30dacf admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:08.479531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24e62a8-fbd0-478c-b1e9-be2c5a30dacf admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:08.480221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24e62a8-fbd0-478c-b1e9-be2c5a30dacf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:08.482261 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24e62a8-fbd0-478c-b1e9-be2c5a30dacf admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:08.552671 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 988/2010] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:08 2022] POST /identity/v3/projects => generated 365 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:02:08.557552 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb0af68-1287-46e8-938c-ee9023ae329e admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:08.570549 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb0af68-1287-46e8-938c-ee9023ae329e admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:08.626322 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:08.671936 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9328879f-2a41-4183-8672-5d95662b5eba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:08.672945 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9328879f-2a41-4183-8672-5d95662b5eba admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:08.674762 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9328879f-2a41-4183-8672-5d95662b5eba admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:08.675274 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9328879f-2a41-4183-8672-5d95662b5eba admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:08.677113 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9328879f-2a41-4183-8672-5d95662b5eba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:08.679620 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9328879f-2a41-4183-8672-5d95662b5eba admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:08.718933 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1023/2011] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:08 2022] POST /identity/v3/users => generated 283 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:08.722634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24e62a8-fbd0-478c-b1e9-be2c5a30dacf admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:08.736889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24e62a8-fbd0-478c-b1e9-be2c5a30dacf admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:08.751976 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654b66ca-febb-4b63-8b35-5c358abc6781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:08.752853 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654b66ca-febb-4b63-8b35-5c358abc6781 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:08.753303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654b66ca-febb-4b63-8b35-5c358abc6781 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:08.753787 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654b66ca-febb-4b63-8b35-5c358abc6781 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:08.754593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654b66ca-febb-4b63-8b35-5c358abc6781 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:08.756570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654b66ca-febb-4b63-8b35-5c358abc6781 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:08.800847 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 989/2012] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:08 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:08.804859 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9328879f-2a41-4183-8672-5d95662b5eba admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:08.823275 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9328879f-2a41-4183-8672-5d95662b5eba admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:08.845032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4441937-a943-4826-8db5-7e6adca38401 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:08.845666 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4441937-a943-4826-8db5-7e6adca38401 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:08.845939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4441937-a943-4826-8db5-7e6adca38401 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:08.846209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4441937-a943-4826-8db5-7e6adca38401 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:08.846859 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4441937-a943-4826-8db5-7e6adca38401 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:08.848480 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4441937-a943-4826-8db5-7e6adca38401 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:08.868302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1024/2013] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:08 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:08.869510 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654b66ca-febb-4b63-8b35-5c358abc6781 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:08.881743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654b66ca-febb-4b63-8b35-5c358abc6781 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:08.889503 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5eaa7f6-e572-4b8a-afd9-bdf0f09ae944 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:08.890191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5eaa7f6-e572-4b8a-afd9-bdf0f09ae944 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:08.890691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5eaa7f6-e572-4b8a-afd9-bdf0f09ae944 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:08.891193 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5eaa7f6-e572-4b8a-afd9-bdf0f09ae944 admin admin] PATH_INFO: `/v3/projects/d1c817502ae14a9fac5b2683608f8a34/users/eb7d4a4482924cdcad44b726c56bc5aa/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:08.910816 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5eaa7f6-e572-4b8a-afd9-bdf0f09ae944 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1c817502ae14a9fac5b2683608f8a34, user_id=eb7d4a4482924cdcad44b726c56bc5aa, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:08.923989 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5eaa7f6-e572-4b8a-afd9-bdf0f09ae944 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:08.961365 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 990/2014] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:08 2022] PUT /identity/v3/projects/d1c817502ae14a9fac5b2683608f8a34/users/eb7d4a4482924cdcad44b726c56bc5aa/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:02:08.975242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e67074-9a86-4be3-9f3f-a79625024ba4 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:08.975580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e67074-9a86-4be3-9f3f-a79625024ba4 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:08.983111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e67074-9a86-4be3-9f3f-a79625024ba4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:09.060634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-02e67074-9a86-4be3-9f3f-a79625024ba4 None None] MFA Rules not processed for user `eb7d4a4482924cdcad44b726c56bc5aa`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:09.062036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-02e67074-9a86-4be3-9f3f-a79625024ba4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:09.165016 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1025/2015] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:08 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:09.183815 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5eaa7f6-e572-4b8a-afd9-bdf0f09ae944 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:09.188478 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a5eaa7f6-e572-4b8a-afd9-bdf0f09ae944 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:09.291042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5eaa7f6-e572-4b8a-afd9-bdf0f09ae944 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:09.296076 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef926454-7555-41e6-bf9e-b95b70b30132 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:09.296904 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef926454-7555-41e6-bf9e-b95b70b30132 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:09.298142 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef926454-7555-41e6-bf9e-b95b70b30132 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:09.298777 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef926454-7555-41e6-bf9e-b95b70b30132 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:09.312220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef926454-7555-41e6-bf9e-b95b70b30132 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:09.313857 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef926454-7555-41e6-bf9e-b95b70b30132 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:09.329030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 991/2016] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:09 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 15:02:10.416816 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e67074-9a86-4be3-9f3f-a79625024ba4 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:10.418355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-02e67074-9a86-4be3-9f3f-a79625024ba4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:10.510211 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e67074-9a86-4be3-9f3f-a79625024ba4 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:10.514305 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e89d41-8db5-4c01-8d4c-713e4ee4f309 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:10.514997 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e89d41-8db5-4c01-8d4c-713e4ee4f309 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:10.515347 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e89d41-8db5-4c01-8d4c-713e4ee4f309 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:10.515725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e89d41-8db5-4c01-8d4c-713e4ee4f309 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:10.517545 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d9e89d41-8db5-4c01-8d4c-713e4ee4f309 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:10.539209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e89d41-8db5-4c01-8d4c-713e4ee4f309 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:10.541448 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e89d41-8db5-4c01-8d4c-713e4ee4f309 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:10.555684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1026/2017] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:10 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:11.441091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef926454-7555-41e6-bf9e-b95b70b30132 service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:11.443507 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ef926454-7555-41e6-bf9e-b95b70b30132 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:11.568918 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:DELETE=5,keystone:SELECT=92,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:11.620432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef926454-7555-41e6-bf9e-b95b70b30132 service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:11.626891 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6f8d26-65d5-4e22-a88b-4464563f7454 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:11.627520 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6f8d26-65d5-4e22-a88b-4464563f7454 service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:11.628130 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6f8d26-65d5-4e22-a88b-4464563f7454 service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:11.628413 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6f8d26-65d5-4e22-a88b-4464563f7454 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:11.637971 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ed6f8d26-65d5-4e22-a88b-4464563f7454 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:11.658144 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6f8d26-65d5-4e22-a88b-4464563f7454 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:11.660490 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6f8d26-65d5-4e22-a88b-4464563f7454 service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:11.696250 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 992/2018] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:18.876145 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e89d41-8db5-4c01-8d4c-713e4ee4f309 service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:18.921240 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:18.943699 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e89d41-8db5-4c01-8d4c-713e4ee4f309 service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:18.948493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d173285-8523-4f80-beb7-26fc3f7420b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:18.949120 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d173285-8523-4f80-beb7-26fc3f7420b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:18.949397 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d173285-8523-4f80-beb7-26fc3f7420b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:18.949667 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d173285-8523-4f80-beb7-26fc3f7420b2 admin admin] PATH_INFO: `/v3/users/eb7d4a4482924cdcad44b726c56bc5aa` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:18.950782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d173285-8523-4f80-beb7-26fc3f7420b2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eb7d4a4482924cdcad44b726c56bc5aa)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:18.952725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d173285-8523-4f80-beb7-26fc3f7420b2 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:19.007261 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-9d173285-8523-4f80-beb7-26fc3f7420b2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eb7d4a4482924cdcad44b726c56bc5aa'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:19.007930 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-9d173285-8523-4f80-beb7-26fc3f7420b2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eb7d4a4482924cdcad44b726c56bc5aa'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:19.022527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-9d173285-8523-4f80-beb7-26fc3f7420b2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eb7d4a4482924cdcad44b726c56bc5aa'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:19.031844 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-9d173285-8523-4f80-beb7-26fc3f7420b2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eb7d4a4482924cdcad44b726c56bc5aa'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:19.040826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-9d173285-8523-4f80-beb7-26fc3f7420b2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eb7d4a4482924cdcad44b726c56bc5aa'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:19.049803 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1027/2019] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:18 2022] DELETE /identity/v3/users/eb7d4a4482924cdcad44b726c56bc5aa => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:19.054408 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021196c0-d0ae-470e-a168-ec354f6eeedc None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:19.054882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021196c0-d0ae-470e-a168-ec354f6eeedc None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:19.054965 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021196c0-d0ae-470e-a168-ec354f6eeedc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:19.074246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-021196c0-d0ae-470e-a168-ec354f6eeedc None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:19.075461 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-021196c0-d0ae-470e-a168-ec354f6eeedc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:19.148355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 993/2020] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:19.155443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d173285-8523-4f80-beb7-26fc3f7420b2 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:19.157063 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-9d173285-8523-4f80-beb7-26fc3f7420b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:19.215788 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d173285-8523-4f80-beb7-26fc3f7420b2 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:19.219381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875b472f-fb0a-401b-8c88-e293529f66fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:19.220007 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875b472f-fb0a-401b-8c88-e293529f66fb service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:19.220287 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875b472f-fb0a-401b-8c88-e293529f66fb service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:19.220558 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875b472f-fb0a-401b-8c88-e293529f66fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:19.230925 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875b472f-fb0a-401b-8c88-e293529f66fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:19.232381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875b472f-fb0a-401b-8c88-e293529f66fb service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:19.241957 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1028/2021] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:19.517776 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021196c0-d0ae-470e-a168-ec354f6eeedc None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:19.522966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-021196c0-d0ae-470e-a168-ec354f6eeedc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:19.538273 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021196c0-d0ae-470e-a168-ec354f6eeedc None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:19.545748 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6cdcd6-2273-44ca-bb8a-405d1556d699 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:19.546469 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6cdcd6-2273-44ca-bb8a-405d1556d699 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:19.546818 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6cdcd6-2273-44ca-bb8a-405d1556d699 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:19.547167 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6cdcd6-2273-44ca-bb8a-405d1556d699 admin admin] PATH_INFO: `/v3/projects/d1c817502ae14a9fac5b2683608f8a34` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:19.548418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6cdcd6-2273-44ca-bb8a-405d1556d699 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d1c817502ae14a9fac5b2683608f8a34)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:19.550407 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6cdcd6-2273-44ca-bb8a-405d1556d699 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:19.621034 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2e6cdcd6-2273-44ca-bb8a-405d1556d699 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd1c817502ae14a9fac5b2683608f8a34', 'request_id': 'req-2e6cdcd6-2273-44ca-bb8a-405d1556d699'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:19.642947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2e6cdcd6-2273-44ca-bb8a-405d1556d699 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd1c817502ae14a9fac5b2683608f8a34', 'request_id': 'req-2e6cdcd6-2273-44ca-bb8a-405d1556d699'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:19.654070 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2e6cdcd6-2273-44ca-bb8a-405d1556d699 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd1c817502ae14a9fac5b2683608f8a34', 'request_id': 'req-2e6cdcd6-2273-44ca-bb8a-405d1556d699'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:19.671257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2e6cdcd6-2273-44ca-bb8a-405d1556d699 admin admin] The token cache is being invalidate because project d1c817502ae14a9fac5b2683608f8a34 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:19.671765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2e6cdcd6-2273-44ca-bb8a-405d1556d699 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:19.672540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2e6cdcd6-2273-44ca-bb8a-405d1556d699 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:19.674332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 994/2022] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:19 2022] DELETE /identity/v3/projects/d1c817502ae14a9fac5b2683608f8a34 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:19.679009 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013a75f4-408e-4223-808a-0c7d9391ecb1 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:19.679357 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013a75f4-408e-4223-808a-0c7d9391ecb1 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:19.679938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013a75f4-408e-4223-808a-0c7d9391ecb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:19.701331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-013a75f4-408e-4223-808a-0c7d9391ecb1 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:19.702640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-013a75f4-408e-4223-808a-0c7d9391ecb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:19.749886 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6cdcd6-2273-44ca-bb8a-405d1556d699 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:19.751248 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2e6cdcd6-2273-44ca-bb8a-405d1556d699 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:19.788164 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1029/2023] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:19.797129 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a227f3c-f90a-481d-9d6c-520de47208c5 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:19.797950 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a227f3c-f90a-481d-9d6c-520de47208c5 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:19.798399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a227f3c-f90a-481d-9d6c-520de47208c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:19.809459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:19.831174 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-6a227f3c-f90a-481d-9d6c-520de47208c5 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:19.832461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-6a227f3c-f90a-481d-9d6c-520de47208c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:19.843844 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1030/2024] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:19.848334 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a227f3c-f90a-481d-9d6c-520de47208c5 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:19.857581 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a227f3c-f90a-481d-9d6c-520de47208c5 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:19.863034 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e28fcae-28c7-4726-b846-f69dcc83d944 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:19.863450 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6cdcd6-2273-44ca-bb8a-405d1556d699 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:19.863689 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e28fcae-28c7-4726-b846-f69dcc83d944 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:19.863976 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e28fcae-28c7-4726-b846-f69dcc83d944 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:19.864255 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e28fcae-28c7-4726-b846-f69dcc83d944 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:19.864910 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e28fcae-28c7-4726-b846-f69dcc83d944 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:02:19.865218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e28fcae-28c7-4726-b846-f69dcc83d944 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:19.866831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e28fcae-28c7-4726-b846-f69dcc83d944 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:19.871222 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e685bc7-4415-49a9-8960-65a0ab5e2397 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:19.871983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e685bc7-4415-49a9-8960-65a0ab5e2397 service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:19.872267 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e685bc7-4415-49a9-8960-65a0ab5e2397 service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:19.872541 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e685bc7-4415-49a9-8960-65a0ab5e2397 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:19.874325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-7e685bc7-4415-49a9-8960-65a0ab5e2397 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:19.880081 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1031/2025] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:02:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 15:02:19.884710 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e28fcae-28c7-4726-b846-f69dcc83d944 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:19.887305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:19.890906 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e685bc7-4415-49a9-8960-65a0ab5e2397 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:19.893004 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e685bc7-4415-49a9-8960-65a0ab5e2397 service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:19.897562 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e28fcae-28c7-4726-b846-f69dcc83d944 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:19.905041 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5ec459-a841-40e6-8446-bc3b92896b82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:19.905670 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5ec459-a841-40e6-8446-bc3b92896b82 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:19.906381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5ec459-a841-40e6-8446-bc3b92896b82 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:19.906667 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5ec459-a841-40e6-8446-bc3b92896b82 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:19.907317 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5ec459-a841-40e6-8446-bc3b92896b82 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:19.908302 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 995/2026] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:19.909995 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5ec459-a841-40e6-8446-bc3b92896b82 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:19.958776 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1032/2027] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:19 2022] POST /identity/v3/projects => generated 405 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:02:19.962973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e685bc7-4415-49a9-8960-65a0ab5e2397 service nova] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:19.975175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e685bc7-4415-49a9-8960-65a0ab5e2397 service nova] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:20.066930 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62a3360-505a-4a3d-aa1d-aaa9d0633dae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:20.067808 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62a3360-505a-4a3d-aa1d-aaa9d0633dae admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:20.067961 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62a3360-505a-4a3d-aa1d-aaa9d0633dae admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:20.068197 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62a3360-505a-4a3d-aa1d-aaa9d0633dae admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:20.068841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62a3360-505a-4a3d-aa1d-aaa9d0633dae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:20.070795 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62a3360-505a-4a3d-aa1d-aaa9d0633dae admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:20.107301 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 996/2028] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:19 2022] POST /identity/v3/users => generated 303 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:20.109626 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5ec459-a841-40e6-8446-bc3b92896b82 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:20.121708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5ec459-a841-40e6-8446-bc3b92896b82 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:20.129616 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92aafba-4658-4ddb-b201-ac9f4a1b10d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:20.130269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92aafba-4658-4ddb-b201-ac9f4a1b10d0 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:20.131191 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92aafba-4658-4ddb-b201-ac9f4a1b10d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:20.131191 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92aafba-4658-4ddb-b201-ac9f4a1b10d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:20.131591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92aafba-4658-4ddb-b201-ac9f4a1b10d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:20.133386 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92aafba-4658-4ddb-b201-ac9f4a1b10d0 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:20.154249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1033/2029] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:20 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:20.155239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62a3360-505a-4a3d-aa1d-aaa9d0633dae admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:20.169645 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62a3360-505a-4a3d-aa1d-aaa9d0633dae admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:20.182790 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e28812-09b7-4c35-9770-64c15df6aa55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:20.183583 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e28812-09b7-4c35-9770-64c15df6aa55 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:20.183878 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e28812-09b7-4c35-9770-64c15df6aa55 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:20.184152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e28812-09b7-4c35-9770-64c15df6aa55 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:20.184812 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e28812-09b7-4c35-9770-64c15df6aa55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:20.186559 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e28812-09b7-4c35-9770-64c15df6aa55 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:20.210037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 997/2030] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:20 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:20.214520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92aafba-4658-4ddb-b201-ac9f4a1b10d0 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:20.226673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92aafba-4658-4ddb-b201-ac9f4a1b10d0 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:20.246832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93372b64-a54e-41e0-b3f8-c146e9663e13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:20.247713 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93372b64-a54e-41e0-b3f8-c146e9663e13 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:20.248069 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93372b64-a54e-41e0-b3f8-c146e9663e13 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:20.248430 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93372b64-a54e-41e0-b3f8-c146e9663e13 admin admin] PATH_INFO: `/v3/projects/0f1eff3e1bda462c924b177b697d2aa4/users/86c57d9980ee41bc97cc4c2044aeb78b/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:20.270036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93372b64-a54e-41e0-b3f8-c146e9663e13 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f1eff3e1bda462c924b177b697d2aa4, user_id=86c57d9980ee41bc97cc4c2044aeb78b, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:20.276050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93372b64-a54e-41e0-b3f8-c146e9663e13 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:20.293279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1034/2031] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:20 2022] PUT /identity/v3/projects/0f1eff3e1bda462c924b177b697d2aa4/users/86c57d9980ee41bc97cc4c2044aeb78b/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:20.298474 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1b4dd5-4b14-4313-8307-4f2419adf288 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:20.298793 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1b4dd5-4b14-4313-8307-4f2419adf288 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:20.299073 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1b4dd5-4b14-4313-8307-4f2419adf288 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:20.324787 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-6a1b4dd5-4b14-4313-8307-4f2419adf288 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:20.326164 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6a1b4dd5-4b14-4313-8307-4f2419adf288 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:20.407209 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 998/2032] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:20.417575 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93372b64-a54e-41e0-b3f8-c146e9663e13 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:20.420114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-93372b64-a54e-41e0-b3f8-c146e9663e13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:20.522399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93372b64-a54e-41e0-b3f8-c146e9663e13 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:20.536477 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147fe99e-403b-4485-8fef-bb21f4320c4c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:20.537120 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147fe99e-403b-4485-8fef-bb21f4320c4c service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:20.540875 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147fe99e-403b-4485-8fef-bb21f4320c4c service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:20.540875 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147fe99e-403b-4485-8fef-bb21f4320c4c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:20.564917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147fe99e-403b-4485-8fef-bb21f4320c4c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:20.566729 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147fe99e-403b-4485-8fef-bb21f4320c4c service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:20.580162 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1035/2033] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:21.604345 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ce8bcd-8483-46b0-b1d6-e9a011984088 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:21.604886 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ce8bcd-8483-46b0-b1d6-e9a011984088 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:21.604886 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ce8bcd-8483-46b0-b1d6-e9a011984088 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:21.606540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 999/2034] 10.222.0.36 () {54 vars in 921 bytes} [Mon Oct 31 15:02:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 31 15:02:21.612391 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f87963-4181-448c-82d4-f770ef751312 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:21.613264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f87963-4181-448c-82d4-f770ef751312 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:21.613911 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f87963-4181-448c-82d4-f770ef751312 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:21.636456 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-72f87963-4181-448c-82d4-f770ef751312 None None] MFA Rules not processed for user `c8cace029a92406f850bb0a4ca07c461`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:21.638138 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-72f87963-4181-448c-82d4-f770ef751312 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:21.694910 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2eefea-7dcc-478a-b74a-eee95d1f8944 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:21.695895 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2eefea-7dcc-478a-b74a-eee95d1f8944 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:21.696910 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2eefea-7dcc-478a-b74a-eee95d1f8944 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:21.745005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-9e2eefea-7dcc-478a-b74a-eee95d1f8944 None None] MFA Rules not processed for user `86c57d9980ee41bc97cc4c2044aeb78b`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:21.746338 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-9e2eefea-7dcc-478a-b74a-eee95d1f8944 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:21.785107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1036/2035] 10.222.0.36 () {58 vars in 1018 bytes} [Mon Oct 31 15:02:21 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:21.802332 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f87963-4181-448c-82d4-f770ef751312 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:21.818436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-72f87963-4181-448c-82d4-f770ef751312 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:21.846985 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:21.859349 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1000/2036] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:21 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:21.866097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2eefea-7dcc-478a-b74a-eee95d1f8944 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:21.875084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:21.880622 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2eefea-7dcc-478a-b74a-eee95d1f8944 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:21.888630 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc8c49e-94f2-4915-b399-1e8633062158 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:21.889299 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc8c49e-94f2-4915-b399-1e8633062158 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:21.889771 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc8c49e-94f2-4915-b399-1e8633062158 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:21.890050 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc8c49e-94f2-4915-b399-1e8633062158 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:21.890703 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc8c49e-94f2-4915-b399-1e8633062158 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:21.892055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f87963-4181-448c-82d4-f770ef751312 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:21.892732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc8c49e-94f2-4915-b399-1e8633062158 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:21.896831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81099e9-9703-4603-88ee-a9a38f4d5007 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9808dcba0b64027bb70e31d1569957d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:21.897620 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81099e9-9703-4603-88ee-a9a38f4d5007 service glance] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:21.897924 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81099e9-9703-4603-88ee-a9a38f4d5007 service glance] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:21.898613 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81099e9-9703-4603-88ee-a9a38f4d5007 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:21.912599 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81099e9-9703-4603-88ee-a9a38f4d5007 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:21.914024 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81099e9-9703-4603-88ee-a9a38f4d5007 service glance] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:21.927589 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1001/2037] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:21 2022] POST /identity/v3/projects => generated 403 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:02:21.928538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc8c49e-94f2-4915-b399-1e8633062158 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:21.929575 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1037/2038] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:21 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 15:02:21.996960 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc8c49e-94f2-4915-b399-1e8633062158 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:22.002061 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bfc8e6-9a9d-4666-b8da-782959f63cdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:22.002672 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bfc8e6-9a9d-4666-b8da-782959f63cdc admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:22.002807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee0d73d-9689-4f4c-b98b-b5ed43f327b8 None None] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:22.003012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bfc8e6-9a9d-4666-b8da-782959f63cdc admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:22.003117 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee0d73d-9689-4f4c-b98b-b5ed43f327b8 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:22.003217 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bfc8e6-9a9d-4666-b8da-782959f63cdc admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:22.003317 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee0d73d-9689-4f4c-b98b-b5ed43f327b8 None None] PATH_INFO: `/` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:22.003883 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bfc8e6-9a9d-4666-b8da-782959f63cdc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:22.004762 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1038/2039] 10.222.0.36 () {54 vars in 921 bytes} [Mon Oct 31 15:02:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 15:02:22.005945 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bfc8e6-9a9d-4666-b8da-782959f63cdc admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:22.009281 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3099da-4957-4f60-8acb-a13a19eb0147 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:22.009582 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3099da-4957-4f60-8acb-a13a19eb0147 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:22.009739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3099da-4957-4f60-8acb-a13a19eb0147 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:22.032005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1002/2040] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:21 2022] POST /identity/v3/users => generated 308 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:22.032209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-1d3099da-4957-4f60-8acb-a13a19eb0147 None None] MFA Rules not processed for user `14946d97fd384e2a8bff68c4f53c2cfe`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:22.032310 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1d3099da-4957-4f60-8acb-a13a19eb0147 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:22.033056 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bfc8e6-9a9d-4666-b8da-782959f63cdc admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:22.044340 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bfc8e6-9a9d-4666-b8da-782959f63cdc admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:22.049420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a623d214-3621-4111-aa09-c13ff66d566a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:22.050027 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a623d214-3621-4111-aa09-c13ff66d566a admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:22.050314 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a623d214-3621-4111-aa09-c13ff66d566a admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:22.050590 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a623d214-3621-4111-aa09-c13ff66d566a admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:22.051236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a623d214-3621-4111-aa09-c13ff66d566a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:22.052952 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a623d214-3621-4111-aa09-c13ff66d566a admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:22.066830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1003/2041] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:22 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:22.067795 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a623d214-3621-4111-aa09-c13ff66d566a admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:22.077897 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a623d214-3621-4111-aa09-c13ff66d566a admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:22.082615 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5223f236-6162-4a29-846a-5605e070abe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:22.083297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5223f236-6162-4a29-846a-5605e070abe5 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:22.083620 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5223f236-6162-4a29-846a-5605e070abe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:22.083977 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5223f236-6162-4a29-846a-5605e070abe5 admin admin] PATH_INFO: `/v3/projects/bff8c53fd3764dfc87714a61c02424dd/users/7b56b8028d474330b5d178ec24e48705/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:22.090055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1039/2042] 10.222.0.36 () {58 vars in 1018 bytes} [Mon Oct 31 15:02:22 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:02:22.101618 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3099da-4957-4f60-8acb-a13a19eb0147 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:22.103937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5223f236-6162-4a29-846a-5605e070abe5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bff8c53fd3764dfc87714a61c02424dd, user_id=7b56b8028d474330b5d178ec24e48705, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:22.106927 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5223f236-6162-4a29-846a-5605e070abe5 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:22.111261 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3099da-4957-4f60-8acb-a13a19eb0147 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:22.115577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8872418-1466-4395-8168-9a4f15fb9a52 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:22.116802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8872418-1466-4395-8168-9a4f15fb9a52 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:22.117096 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8872418-1466-4395-8168-9a4f15fb9a52 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:22.117373 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8872418-1466-4395-8168-9a4f15fb9a52 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:22.121629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1004/2043] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:22 2022] PUT /identity/v3/projects/bff8c53fd3764dfc87714a61c02424dd/users/7b56b8028d474330b5d178ec24e48705/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:02:22.126219 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5223f236-6162-4a29-846a-5605e070abe5 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:22.128414 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8872418-1466-4395-8168-9a4f15fb9a52 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:22.130054 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8872418-1466-4395-8168-9a4f15fb9a52 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:22.151030 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1040/2044] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:22 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 15:02:22.199825 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5223f236-6162-4a29-846a-5605e070abe5 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:22.201528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8b01c4-9dbb-477a-976c-456b4fcd3290 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:22.202241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8b01c4-9dbb-477a-976c-456b4fcd3290 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:22.202688 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8b01c4-9dbb-477a-976c-456b4fcd3290 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:22.203083 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8b01c4-9dbb-477a-976c-456b4fcd3290 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:22.203951 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8b01c4-9dbb-477a-976c-456b4fcd3290 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:22.205642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8b01c4-9dbb-477a-976c-456b4fcd3290 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:22.220052 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1005/2045] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:22 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:22.224217 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8872418-1466-4395-8168-9a4f15fb9a52 service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:22.238313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8872418-1466-4395-8168-9a4f15fb9a52 service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:22.243058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a86841-2e1b-45a7-9e1a-1137f742d43f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:22.244527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a86841-2e1b-45a7-9e1a-1137f742d43f admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:22.244527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a86841-2e1b-45a7-9e1a-1137f742d43f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:22.244768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a86841-2e1b-45a7-9e1a-1137f742d43f admin admin] PATH_INFO: `/v3/projects/bff8c53fd3764dfc87714a61c02424dd/users/7b56b8028d474330b5d178ec24e48705/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:22.245970 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5d699c-d683-4256-b6d8-99f717522745 None None] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:22.246254 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5d699c-d683-4256-b6d8-99f717522745 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:22.246527 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5d699c-d683-4256-b6d8-99f717522745 None None] PATH_INFO: `/` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:22.246935 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a86841-2e1b-45a7-9e1a-1137f742d43f admin admin] RBAC: Authorizing `identity:create_grant(project_id=bff8c53fd3764dfc87714a61c02424dd, user_id=7b56b8028d474330b5d178ec24e48705, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:22.248095 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1006/2046] 10.222.0.36 () {54 vars in 921 bytes} [Mon Oct 31 15:02:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 31 15:02:22.249843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a86841-2e1b-45a7-9e1a-1137f742d43f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:22.252648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40769162-8d04-4871-9c8c-8d8c9c9a1e04 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:22.252950 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40769162-8d04-4871-9c8c-8d8c9c9a1e04 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:22.255726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40769162-8d04-4871-9c8c-8d8c9c9a1e04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:22.263546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1041/2047] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:22 2022] PUT /identity/v3/projects/bff8c53fd3764dfc87714a61c02424dd/users/7b56b8028d474330b5d178ec24e48705/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:02:22.279651 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-40769162-8d04-4871-9c8c-8d8c9c9a1e04 None None] MFA Rules not processed for user `c8cace029a92406f850bb0a4ca07c461`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:22.280887 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-40769162-8d04-4871-9c8c-8d8c9c9a1e04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:22.323266 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:22.379695 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1007/2048] 10.222.0.36 () {58 vars in 1018 bytes} [Mon Oct 31 15:02:22 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:02:22.393522 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a86841-2e1b-45a7-9e1a-1137f742d43f admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:22.395523 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-21a86841-2e1b-45a7-9e1a-1137f742d43f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:22.471333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a86841-2e1b-45a7-9e1a-1137f742d43f admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:22.475455 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cfd005-4821-48d9-be3b-bb55637b2dc6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:22.476161 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cfd005-4821-48d9-be3b-bb55637b2dc6 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:22.476537 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cfd005-4821-48d9-be3b-bb55637b2dc6 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:22.476856 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cfd005-4821-48d9-be3b-bb55637b2dc6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:22.491005 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cfd005-4821-48d9-be3b-bb55637b2dc6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:22.492735 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cfd005-4821-48d9-be3b-bb55637b2dc6 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:22.505239 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1042/2049] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:22 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:23.032805 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc96072-e50f-441f-b5a2-e96e00afbdd2 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:23.033298 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc96072-e50f-441f-b5a2-e96e00afbdd2 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:23.033554 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc96072-e50f-441f-b5a2-e96e00afbdd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:23.062465 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-4dc96072-e50f-441f-b5a2-e96e00afbdd2 None None] MFA Rules not processed for user `7b56b8028d474330b5d178ec24e48705`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:23.063833 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-4dc96072-e50f-441f-b5a2-e96e00afbdd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:23.139954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1008/2050] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:23 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:02:24.224285 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cfd005-4821-48d9-be3b-bb55637b2dc6 service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:24.275399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:24.291642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cfd005-4821-48d9-be3b-bb55637b2dc6 service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:24.296390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5232fafa-1924-4c8f-b5fa-9ffe4a20b197 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:24.297020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5232fafa-1924-4c8f-b5fa-9ffe4a20b197 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:24.297300 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5232fafa-1924-4c8f-b5fa-9ffe4a20b197 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:24.297581 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5232fafa-1924-4c8f-b5fa-9ffe4a20b197 admin admin] PATH_INFO: `/v3/users/86c57d9980ee41bc97cc4c2044aeb78b` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:24.298790 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5232fafa-1924-4c8f-b5fa-9ffe4a20b197 admin admin] RBAC: Authorizing `identity:delete_user(user_id=86c57d9980ee41bc97cc4c2044aeb78b)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:24.300874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5232fafa-1924-4c8f-b5fa-9ffe4a20b197 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:24.352593 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5232fafa-1924-4c8f-b5fa-9ffe4a20b197 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '86c57d9980ee41bc97cc4c2044aeb78b'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:24.353247 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5232fafa-1924-4c8f-b5fa-9ffe4a20b197 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '86c57d9980ee41bc97cc4c2044aeb78b'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:24.367257 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5232fafa-1924-4c8f-b5fa-9ffe4a20b197 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '86c57d9980ee41bc97cc4c2044aeb78b'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:24.375499 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5232fafa-1924-4c8f-b5fa-9ffe4a20b197 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '86c57d9980ee41bc97cc4c2044aeb78b'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:24.386096 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5232fafa-1924-4c8f-b5fa-9ffe4a20b197 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '86c57d9980ee41bc97cc4c2044aeb78b'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:24.397088 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1043/2051] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:24 2022] DELETE /identity/v3/users/86c57d9980ee41bc97cc4c2044aeb78b => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:24.588710 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc96072-e50f-441f-b5a2-e96e00afbdd2 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:24.603268 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc96072-e50f-441f-b5a2-e96e00afbdd2 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:24.665472 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a204b54-1785-44db-8139-a5436eb0b0bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:24.666114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a204b54-1785-44db-8139-a5436eb0b0bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:24.666508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a204b54-1785-44db-8139-a5436eb0b0bf admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:24.666789 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a204b54-1785-44db-8139-a5436eb0b0bf admin admin] PATH_INFO: `/v3/projects/0f1eff3e1bda462c924b177b697d2aa4` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:24.667969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a204b54-1785-44db-8139-a5436eb0b0bf admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f1eff3e1bda462c924b177b697d2aa4)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:24.669879 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a204b54-1785-44db-8139-a5436eb0b0bf admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:24.719009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-3a204b54-1785-44db-8139-a5436eb0b0bf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f1eff3e1bda462c924b177b697d2aa4', 'request_id': 'req-3a204b54-1785-44db-8139-a5436eb0b0bf'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:24.722009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:24.729746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-3a204b54-1785-44db-8139-a5436eb0b0bf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f1eff3e1bda462c924b177b697d2aa4', 'request_id': 'req-3a204b54-1785-44db-8139-a5436eb0b0bf'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:24.736558 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-3a204b54-1785-44db-8139-a5436eb0b0bf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f1eff3e1bda462c924b177b697d2aa4', 'request_id': 'req-3a204b54-1785-44db-8139-a5436eb0b0bf'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:24.747956 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-3a204b54-1785-44db-8139-a5436eb0b0bf admin admin] The token cache is being invalidate because project 0f1eff3e1bda462c924b177b697d2aa4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:24.748257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-3a204b54-1785-44db-8139-a5436eb0b0bf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:24.752017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-3a204b54-1785-44db-8139-a5436eb0b0bf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:24.754378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1009/2052] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:24 2022] DELETE /identity/v3/projects/0f1eff3e1bda462c924b177b697d2aa4 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:24.758980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5232fafa-1924-4c8f-b5fa-9ffe4a20b197 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:24.760845 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5232fafa-1924-4c8f-b5fa-9ffe4a20b197 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:24.774964 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5232fafa-1924-4c8f-b5fa-9ffe4a20b197 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:24.833572 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72aa9d27-0f10-436f-bca3-8b2a8c4baaeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:24.834203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72aa9d27-0f10-436f-bca3-8b2a8c4baaeb admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:24.834485 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72aa9d27-0f10-436f-bca3-8b2a8c4baaeb admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:24.834758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72aa9d27-0f10-436f-bca3-8b2a8c4baaeb admin admin] PATH_INFO: `/v3/users/7b56b8028d474330b5d178ec24e48705` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:24.836225 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72aa9d27-0f10-436f-bca3-8b2a8c4baaeb admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b56b8028d474330b5d178ec24e48705)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:24.838976 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72aa9d27-0f10-436f-bca3-8b2a8c4baaeb admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:24.889399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-72aa9d27-0f10-436f-bca3-8b2a8c4baaeb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b56b8028d474330b5d178ec24e48705'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:24.890163 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-72aa9d27-0f10-436f-bca3-8b2a8c4baaeb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b56b8028d474330b5d178ec24e48705'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:24.902894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-72aa9d27-0f10-436f-bca3-8b2a8c4baaeb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b56b8028d474330b5d178ec24e48705'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:24.907339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=80,keystone:DELETE=19,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:24.910795 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-72aa9d27-0f10-436f-bca3-8b2a8c4baaeb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b56b8028d474330b5d178ec24e48705'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:24.919943 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-72aa9d27-0f10-436f-bca3-8b2a8c4baaeb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b56b8028d474330b5d178ec24e48705'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:24.926319 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1044/2053] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:24 2022] DELETE /identity/v3/users/7b56b8028d474330b5d178ec24e48705 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:25.100703 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a204b54-1785-44db-8139-a5436eb0b0bf admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:25.102365 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-3a204b54-1785-44db-8139-a5436eb0b0bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:25.133139 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a204b54-1785-44db-8139-a5436eb0b0bf admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:25.207600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5398b43-323c-483d-b653-e2ae79e4903c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:25.208381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5398b43-323c-483d-b653-e2ae79e4903c admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:25.208766 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5398b43-323c-483d-b653-e2ae79e4903c admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:25.209143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5398b43-323c-483d-b653-e2ae79e4903c admin admin] PATH_INFO: `/v3/projects/bff8c53fd3764dfc87714a61c02424dd` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:25.210508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5398b43-323c-483d-b653-e2ae79e4903c admin admin] RBAC: Authorizing `identity:delete_project(project_id=bff8c53fd3764dfc87714a61c02424dd)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:25.212854 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5398b43-323c-483d-b653-e2ae79e4903c admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:25.271124 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b5398b43-323c-483d-b653-e2ae79e4903c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bff8c53fd3764dfc87714a61c02424dd', 'request_id': 'req-b5398b43-323c-483d-b653-e2ae79e4903c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:25.282111 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b5398b43-323c-483d-b653-e2ae79e4903c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bff8c53fd3764dfc87714a61c02424dd', 'request_id': 'req-b5398b43-323c-483d-b653-e2ae79e4903c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:25.288866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b5398b43-323c-483d-b653-e2ae79e4903c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bff8c53fd3764dfc87714a61c02424dd', 'request_id': 'req-b5398b43-323c-483d-b653-e2ae79e4903c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:25.302412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b5398b43-323c-483d-b653-e2ae79e4903c admin admin] The token cache is being invalidate because project bff8c53fd3764dfc87714a61c02424dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:25.302702 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b5398b43-323c-483d-b653-e2ae79e4903c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:25.303357 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b5398b43-323c-483d-b653-e2ae79e4903c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:25.305551 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1010/2054] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:25 2022] DELETE /identity/v3/projects/bff8c53fd3764dfc87714a61c02424dd => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:25.311215 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ca2222-b065-4863-8423-c3f5a7a8a44f None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:25.312114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ca2222-b065-4863-8423-c3f5a7a8a44f None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:25.312395 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ca2222-b065-4863-8423-c3f5a7a8a44f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:25.339715 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-f1ca2222-b065-4863-8423-c3f5a7a8a44f None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:25.341068 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-f1ca2222-b065-4863-8423-c3f5a7a8a44f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:25.408197 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1045/2055] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:25.416850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc9e9e1-27cf-4d7d-96ba-f3e07f6873fe None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:25.417134 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc9e9e1-27cf-4d7d-96ba-f3e07f6873fe None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:25.417412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc9e9e1-27cf-4d7d-96ba-f3e07f6873fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:25.438916 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-ffc9e9e1-27cf-4d7d-96ba-f3e07f6873fe None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:25.440221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ffc9e9e1-27cf-4d7d-96ba-f3e07f6873fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:25.449510 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1011/2056] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:25.453833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ca2222-b065-4863-8423-c3f5a7a8a44f None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:25.464733 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ca2222-b065-4863-8423-c3f5a7a8a44f None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:25.469430 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96f6d5a-166e-4c20-a8ed-4446471ee875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:25.470048 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f6d5a-166e-4c20-a8ed-4446471ee875 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:25.470333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f6d5a-166e-4c20-a8ed-4446471ee875 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:25.470610 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f6d5a-166e-4c20-a8ed-4446471ee875 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:25.471267 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96f6d5a-166e-4c20-a8ed-4446471ee875 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:02:25.471593 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96f6d5a-166e-4c20-a8ed-4446471ee875 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:25.473224 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96f6d5a-166e-4c20-a8ed-4446471ee875 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:25.485925 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1046/2057] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:02:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:02:25.490319 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc9e9e1-27cf-4d7d-96ba-f3e07f6873fe None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:25.501616 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc9e9e1-27cf-4d7d-96ba-f3e07f6873fe None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:25.509432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152ddcae-a829-4683-946d-aa5d1937579f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:25.512007 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152ddcae-a829-4683-946d-aa5d1937579f admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:25.513616 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152ddcae-a829-4683-946d-aa5d1937579f admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:25.513616 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152ddcae-a829-4683-946d-aa5d1937579f admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:25.513616 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152ddcae-a829-4683-946d-aa5d1937579f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:25.517286 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152ddcae-a829-4683-946d-aa5d1937579f admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:25.551803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1012/2058] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:25 2022] POST /identity/v3/projects => generated 365 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:25.556056 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96f6d5a-166e-4c20-a8ed-4446471ee875 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:25.567883 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96f6d5a-166e-4c20-a8ed-4446471ee875 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:25.634834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5d4ef2-b4bf-4405-b4c4-8c96b8f68132 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:25.635546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5d4ef2-b4bf-4405-b4c4-8c96b8f68132 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:25.635978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5d4ef2-b4bf-4405-b4c4-8c96b8f68132 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:25.636283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5d4ef2-b4bf-4405-b4c4-8c96b8f68132 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:25.637056 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5d4ef2-b4bf-4405-b4c4-8c96b8f68132 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:25.639137 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5d4ef2-b4bf-4405-b4c4-8c96b8f68132 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:25.663849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1047/2059] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:25 2022] POST /identity/v3/users => generated 283 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:25.667894 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152ddcae-a829-4683-946d-aa5d1937579f admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:25.679814 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152ddcae-a829-4683-946d-aa5d1937579f admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:25.686414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf28b77-5650-491c-91ef-f557887366a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:25.687134 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf28b77-5650-491c-91ef-f557887366a5 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:25.687417 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf28b77-5650-491c-91ef-f557887366a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:25.687872 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf28b77-5650-491c-91ef-f557887366a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:25.688530 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf28b77-5650-491c-91ef-f557887366a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:25.690150 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf28b77-5650-491c-91ef-f557887366a5 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:25.707290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1013/2060] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:25 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:25.711204 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5d4ef2-b4bf-4405-b4c4-8c96b8f68132 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:25.721747 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5d4ef2-b4bf-4405-b4c4-8c96b8f68132 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:25.726339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488766ff-6e57-4642-948c-8112550add75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:25.727020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488766ff-6e57-4642-948c-8112550add75 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:25.727395 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488766ff-6e57-4642-948c-8112550add75 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:25.727704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488766ff-6e57-4642-948c-8112550add75 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:25.728401 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488766ff-6e57-4642-948c-8112550add75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:25.729992 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488766ff-6e57-4642-948c-8112550add75 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:25.741076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1048/2061] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:25.746194 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf28b77-5650-491c-91ef-f557887366a5 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:25.756903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf28b77-5650-491c-91ef-f557887366a5 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:25.762552 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3665f0-ff2d-4bf3-b57f-9347ddf6a4bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:25.763220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3665f0-ff2d-4bf3-b57f-9347ddf6a4bb admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:25.763504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3665f0-ff2d-4bf3-b57f-9347ddf6a4bb admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:25.763874 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3665f0-ff2d-4bf3-b57f-9347ddf6a4bb admin admin] PATH_INFO: `/v3/projects/2709e9b7affe46acb74a8943b2330599/users/a16fed5fc66b49018ed609a3c94eb6c5/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:25.779540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3665f0-ff2d-4bf3-b57f-9347ddf6a4bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=2709e9b7affe46acb74a8943b2330599, user_id=a16fed5fc66b49018ed609a3c94eb6c5, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:25.782329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3665f0-ff2d-4bf3-b57f-9347ddf6a4bb admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:25.796704 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1014/2062] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:25 2022] PUT /identity/v3/projects/2709e9b7affe46acb74a8943b2330599/users/a16fed5fc66b49018ed609a3c94eb6c5/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:02:25.808721 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e9168e-35da-4f5a-b359-e3e710ad2dfa None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:25.809010 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e9168e-35da-4f5a-b359-e3e710ad2dfa None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:25.809288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e9168e-35da-4f5a-b359-e3e710ad2dfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:25.839445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-97e9168e-35da-4f5a-b359-e3e710ad2dfa None None] MFA Rules not processed for user `a16fed5fc66b49018ed609a3c94eb6c5`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:25.840786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-97e9168e-35da-4f5a-b359-e3e710ad2dfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:25.887023 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:25.917810 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1049/2063] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:25 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:25.919112 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3665f0-ff2d-4bf3-b57f-9347ddf6a4bb admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:25.928664 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:DELETE=10,keystone:SELECT=87,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:26.022786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3665f0-ff2d-4bf3-b57f-9347ddf6a4bb admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:26.029999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b1fbec-8545-4e2c-a13c-c6aa6f70c76e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:26.031044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b1fbec-8545-4e2c-a13c-c6aa6f70c76e admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:26.031334 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b1fbec-8545-4e2c-a13c-c6aa6f70c76e admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:26.031660 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b1fbec-8545-4e2c-a13c-c6aa6f70c76e admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:26.033755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b1fbec-8545-4e2c-a13c-c6aa6f70c76e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:26.035749 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b1fbec-8545-4e2c-a13c-c6aa6f70c76e admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:26.080450 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1015/2064] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:25 2022] POST /identity/v3/projects => generated 363 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:26.084399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e9168e-35da-4f5a-b359-e3e710ad2dfa None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:26.096458 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e9168e-35da-4f5a-b359-e3e710ad2dfa None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:26.159096 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65814c2-864e-47eb-9cb6-17e0e1b4989f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:26.159793 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65814c2-864e-47eb-9cb6-17e0e1b4989f admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:26.160074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65814c2-864e-47eb-9cb6-17e0e1b4989f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:26.160346 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65814c2-864e-47eb-9cb6-17e0e1b4989f admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:26.160988 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65814c2-864e-47eb-9cb6-17e0e1b4989f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:26.162954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65814c2-864e-47eb-9cb6-17e0e1b4989f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:26.187790 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1050/2065] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:26 2022] POST /identity/v3/users => generated 288 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:02:26.188995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b1fbec-8545-4e2c-a13c-c6aa6f70c76e admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:26.201719 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b1fbec-8545-4e2c-a13c-c6aa6f70c76e admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:26.208653 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ceb4e2-6551-4e72-b3ee-6a4e90cf5094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:26.209346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ceb4e2-6551-4e72-b3ee-6a4e90cf5094 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:26.209692 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ceb4e2-6551-4e72-b3ee-6a4e90cf5094 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:26.210152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ceb4e2-6551-4e72-b3ee-6a4e90cf5094 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:26.212331 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ceb4e2-6551-4e72-b3ee-6a4e90cf5094 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:26.214980 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ceb4e2-6551-4e72-b3ee-6a4e90cf5094 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:26.239074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1016/2066] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:26 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:26.244877 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65814c2-864e-47eb-9cb6-17e0e1b4989f admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:26.257912 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65814c2-864e-47eb-9cb6-17e0e1b4989f admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:26.267831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79c1571-4c8c-409a-9f04-c87291407f0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:26.268583 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79c1571-4c8c-409a-9f04-c87291407f0a admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:26.270304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79c1571-4c8c-409a-9f04-c87291407f0a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:26.270664 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79c1571-4c8c-409a-9f04-c87291407f0a admin admin] PATH_INFO: `/v3/projects/7dd4ddd5553949d4ad242cf3020c60cc/users/14f3288644994db7badac9777d37518e/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:26.293132 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd83956-9381-4bb1-bbe0-3d32050a4ffe None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:26.294011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd83956-9381-4bb1-bbe0-3d32050a4ffe None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:26.294011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd83956-9381-4bb1-bbe0-3d32050a4ffe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:26.306779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79c1571-4c8c-409a-9f04-c87291407f0a admin admin] RBAC: Authorizing `identity:create_grant(project_id=7dd4ddd5553949d4ad242cf3020c60cc, user_id=14f3288644994db7badac9777d37518e, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:26.310242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79c1571-4c8c-409a-9f04-c87291407f0a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:26.319089 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-cbd83956-9381-4bb1-bbe0-3d32050a4ffe None None] MFA Rules not processed for user `e0d0f8598fac40449bcb2a55ff8df8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:26.320702 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-cbd83956-9381-4bb1-bbe0-3d32050a4ffe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:26.346060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1051/2067] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:26 2022] PUT /identity/v3/projects/7dd4ddd5553949d4ad242cf3020c60cc/users/14f3288644994db7badac9777d37518e/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:26.347091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79c1571-4c8c-409a-9f04-c87291407f0a admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:26.404865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1017/2068] 10.222.0.36 () {58 vars in 1009 bytes} [Mon Oct 31 15:02:26 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:26.434979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd83956-9381-4bb1-bbe0-3d32050a4ffe None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:26.436759 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-cbd83956-9381-4bb1-bbe0-3d32050a4ffe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:26.447974 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79c1571-4c8c-409a-9f04-c87291407f0a admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:26.452841 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddcaa8c-37fe-4321-b206-85d895096378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:26.453689 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddcaa8c-37fe-4321-b206-85d895096378 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:26.454257 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddcaa8c-37fe-4321-b206-85d895096378 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:26.454966 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddcaa8c-37fe-4321-b206-85d895096378 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:26.455369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddcaa8c-37fe-4321-b206-85d895096378 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:26.457509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddcaa8c-37fe-4321-b206-85d895096378 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:26.472085 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1052/2069] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:26 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:26.476079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddcaa8c-37fe-4321-b206-85d895096378 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:26.487361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddcaa8c-37fe-4321-b206-85d895096378 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:26.492701 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9821222c-0bb7-44b4-8e1a-1c75bc00b8c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:26.493697 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9821222c-0bb7-44b4-8e1a-1c75bc00b8c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:26.494272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9821222c-0bb7-44b4-8e1a-1c75bc00b8c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:26.494425 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9821222c-0bb7-44b4-8e1a-1c75bc00b8c9 admin admin] PATH_INFO: `/v3/projects/7dd4ddd5553949d4ad242cf3020c60cc/users/14f3288644994db7badac9777d37518e/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:26.496381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9821222c-0bb7-44b4-8e1a-1c75bc00b8c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7dd4ddd5553949d4ad242cf3020c60cc, user_id=14f3288644994db7badac9777d37518e, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:26.499543 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9821222c-0bb7-44b4-8e1a-1c75bc00b8c9 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:26.511768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1053/2070] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:26 2022] PUT /identity/v3/projects/7dd4ddd5553949d4ad242cf3020c60cc/users/14f3288644994db7badac9777d37518e/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:02:26.521864 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16ab4c0-e46e-431d-96d8-3acdc5a43c75 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:26.522140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16ab4c0-e46e-431d-96d8-3acdc5a43c75 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:26.522414 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16ab4c0-e46e-431d-96d8-3acdc5a43c75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:26.523476 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd83956-9381-4bb1-bbe0-3d32050a4ffe None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:26.557947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-d16ab4c0-e46e-431d-96d8-3acdc5a43c75 None None] MFA Rules not processed for user `14f3288644994db7badac9777d37518e`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:26.559618 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d16ab4c0-e46e-431d-96d8-3acdc5a43c75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:26.563921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:26.586168 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:26.619721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a265a574-f9ad-4c78-a293-01d18c388c0a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4ae5e20cc5b40ccbdb951f908f5080d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:26.619721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a265a574-f9ad-4c78-a293-01d18c388c0a service swift] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:26.620069 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a265a574-f9ad-4c78-a293-01d18c388c0a service swift] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:26.620294 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a265a574-f9ad-4c78-a293-01d18c388c0a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:26.667683 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1054/2071] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:26 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:26.681057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16ab4c0-e46e-431d-96d8-3acdc5a43c75 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:26.683288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d16ab4c0-e46e-431d-96d8-3acdc5a43c75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:26.705948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a265a574-f9ad-4c78-a293-01d18c388c0a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:26.707874 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a265a574-f9ad-4c78-a293-01d18c388c0a service swift] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:26.717346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1018/2072] 10.222.0.36 () {60 vars in 1384 bytes} [Mon Oct 31 15:02:26 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 283 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 31 15:02:26.767621 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16ab4c0-e46e-431d-96d8-3acdc5a43c75 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:26.771427 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d74b52f-eaa4-4a7f-92bf-c29f6e6927ca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:26.772075 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d74b52f-eaa4-4a7f-92bf-c29f6e6927ca service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:26.772352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d74b52f-eaa4-4a7f-92bf-c29f6e6927ca service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:26.772647 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d74b52f-eaa4-4a7f-92bf-c29f6e6927ca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:26.783320 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d74b52f-eaa4-4a7f-92bf-c29f6e6927ca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:26.784831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d74b52f-eaa4-4a7f-92bf-c29f6e6927ca service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:26.797159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1055/2073] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:26 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:26.917087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a265a574-f9ad-4c78-a293-01d18c388c0a service swift] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:26.932055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a265a574-f9ad-4c78-a293-01d18c388c0a service swift] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:26.936746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5efc118-584b-4e3f-926e-c288ec0275c2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:26.937492 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5efc118-584b-4e3f-926e-c288ec0275c2 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:26.937889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5efc118-584b-4e3f-926e-c288ec0275c2 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:26.938333 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5efc118-584b-4e3f-926e-c288ec0275c2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:27.051127 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5efc118-584b-4e3f-926e-c288ec0275c2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:27.061592 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5efc118-584b-4e3f-926e-c288ec0275c2 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:27.089219 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1019/2074] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:26 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 15:02:30.025318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d74b52f-eaa4-4a7f-92bf-c29f6e6927ca service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:30.115264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d74b52f-eaa4-4a7f-92bf-c29f6e6927ca service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:30.122103 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a56530-e34d-4a2f-9a5e-4e85bee434c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:30.122866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a56530-e34d-4a2f-9a5e-4e85bee434c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:30.123152 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a56530-e34d-4a2f-9a5e-4e85bee434c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:30.123430 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a56530-e34d-4a2f-9a5e-4e85bee434c3 admin admin] PATH_INFO: `/v3/users/a16fed5fc66b49018ed609a3c94eb6c5` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:30.124778 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a56530-e34d-4a2f-9a5e-4e85bee434c3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a16fed5fc66b49018ed609a3c94eb6c5)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:30.127017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a56530-e34d-4a2f-9a5e-4e85bee434c3 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:30.183047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f7a56530-e34d-4a2f-9a5e-4e85bee434c3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a16fed5fc66b49018ed609a3c94eb6c5'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:30.183746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f7a56530-e34d-4a2f-9a5e-4e85bee434c3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a16fed5fc66b49018ed609a3c94eb6c5'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:30.202063 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:30.213065 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f7a56530-e34d-4a2f-9a5e-4e85bee434c3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a16fed5fc66b49018ed609a3c94eb6c5'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:30.219980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f7a56530-e34d-4a2f-9a5e-4e85bee434c3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a16fed5fc66b49018ed609a3c94eb6c5'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:30.235872 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f7a56530-e34d-4a2f-9a5e-4e85bee434c3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a16fed5fc66b49018ed609a3c94eb6c5'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:30.254026 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1056/2075] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:30 2022] DELETE /identity/v3/users/a16fed5fc66b49018ed609a3c94eb6c5 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:02:30.259463 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12428261-ac52-4af7-9d8a-384c4576f3d4 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:30.260493 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12428261-ac52-4af7-9d8a-384c4576f3d4 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:30.261021 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12428261-ac52-4af7-9d8a-384c4576f3d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:30.286995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-12428261-ac52-4af7-9d8a-384c4576f3d4 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:30.288273 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-12428261-ac52-4af7-9d8a-384c4576f3d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:30.355388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1020/2076] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:30.365132 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a56530-e34d-4a2f-9a5e-4e85bee434c3 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:30.366531 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-f7a56530-e34d-4a2f-9a5e-4e85bee434c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:30.423336 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a56530-e34d-4a2f-9a5e-4e85bee434c3 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:30.426993 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32229f99-fce5-4fca-9d37-4de3b61dd635 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:30.427612 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32229f99-fce5-4fca-9d37-4de3b61dd635 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:30.427893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32229f99-fce5-4fca-9d37-4de3b61dd635 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:30.428168 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32229f99-fce5-4fca-9d37-4de3b61dd635 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:30.437769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32229f99-fce5-4fca-9d37-4de3b61dd635 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:30.438641 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32229f99-fce5-4fca-9d37-4de3b61dd635 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:30.449720 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1057/2077] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:30.749157 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12428261-ac52-4af7-9d8a-384c4576f3d4 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:30.750809 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-12428261-ac52-4af7-9d8a-384c4576f3d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:30.766798 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12428261-ac52-4af7-9d8a-384c4576f3d4 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:30.772077 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9338029-bcc3-4af4-9283-d443817578a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:30.772721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9338029-bcc3-4af4-9283-d443817578a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:30.773009 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9338029-bcc3-4af4-9283-d443817578a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:30.773271 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9338029-bcc3-4af4-9283-d443817578a6 admin admin] PATH_INFO: `/v3/projects/2709e9b7affe46acb74a8943b2330599` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:30.775469 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9338029-bcc3-4af4-9283-d443817578a6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2709e9b7affe46acb74a8943b2330599)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:30.799124 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9338029-bcc3-4af4-9283-d443817578a6 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:30.806872 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:30.864724 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c9338029-bcc3-4af4-9283-d443817578a6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2709e9b7affe46acb74a8943b2330599', 'request_id': 'req-c9338029-bcc3-4af4-9283-d443817578a6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:30.874157 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c9338029-bcc3-4af4-9283-d443817578a6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2709e9b7affe46acb74a8943b2330599', 'request_id': 'req-c9338029-bcc3-4af4-9283-d443817578a6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:30.881232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c9338029-bcc3-4af4-9283-d443817578a6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2709e9b7affe46acb74a8943b2330599', 'request_id': 'req-c9338029-bcc3-4af4-9283-d443817578a6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:30.893702 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c9338029-bcc3-4af4-9283-d443817578a6 admin admin] The token cache is being invalidate because project 2709e9b7affe46acb74a8943b2330599 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:30.894014 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c9338029-bcc3-4af4-9283-d443817578a6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:30.894660 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c9338029-bcc3-4af4-9283-d443817578a6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:30.896262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1021/2078] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:30 2022] DELETE /identity/v3/projects/2709e9b7affe46acb74a8943b2330599 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:30.900894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32229f99-fce5-4fca-9d37-4de3b61dd635 service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:30.917551 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32229f99-fce5-4fca-9d37-4de3b61dd635 service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:30.998526 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c3d662-b937-46ff-84b0-ebec1903421e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:30.999265 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c3d662-b937-46ff-84b0-ebec1903421e admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:30.999601 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c3d662-b937-46ff-84b0-ebec1903421e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:30.999857 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c3d662-b937-46ff-84b0-ebec1903421e admin admin] PATH_INFO: `/v3/users/14f3288644994db7badac9777d37518e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:31.001107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c3d662-b937-46ff-84b0-ebec1903421e admin admin] RBAC: Authorizing `identity:delete_user(user_id=14f3288644994db7badac9777d37518e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:31.002880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c3d662-b937-46ff-84b0-ebec1903421e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:31.052228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-01c3d662-b937-46ff-84b0-ebec1903421e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '14f3288644994db7badac9777d37518e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:31.052889 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-01c3d662-b937-46ff-84b0-ebec1903421e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '14f3288644994db7badac9777d37518e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:31.066761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-01c3d662-b937-46ff-84b0-ebec1903421e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '14f3288644994db7badac9777d37518e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:31.075360 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-01c3d662-b937-46ff-84b0-ebec1903421e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '14f3288644994db7badac9777d37518e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:31.085421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-01c3d662-b937-46ff-84b0-ebec1903421e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '14f3288644994db7badac9777d37518e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:31.088726 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:31.095342 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1058/2079] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:30 2022] DELETE /identity/v3/users/14f3288644994db7badac9777d37518e => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:02:31.451428 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9338029-bcc3-4af4-9283-d443817578a6 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:31.453763 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-c9338029-bcc3-4af4-9283-d443817578a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:31.469514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9338029-bcc3-4af4-9283-d443817578a6 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:31.549667 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1399bf60-8f9b-471d-85d6-f791c646b76c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:31.550331 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1399bf60-8f9b-471d-85d6-f791c646b76c admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:31.550636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1399bf60-8f9b-471d-85d6-f791c646b76c admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:31.550907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1399bf60-8f9b-471d-85d6-f791c646b76c admin admin] PATH_INFO: `/v3/projects/7dd4ddd5553949d4ad242cf3020c60cc` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:31.552047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1399bf60-8f9b-471d-85d6-f791c646b76c admin admin] RBAC: Authorizing `identity:delete_project(project_id=7dd4ddd5553949d4ad242cf3020c60cc)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:31.553951 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1399bf60-8f9b-471d-85d6-f791c646b76c admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:31.620094 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-1399bf60-8f9b-471d-85d6-f791c646b76c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7dd4ddd5553949d4ad242cf3020c60cc', 'request_id': 'req-1399bf60-8f9b-471d-85d6-f791c646b76c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:31.628354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-1399bf60-8f9b-471d-85d6-f791c646b76c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7dd4ddd5553949d4ad242cf3020c60cc', 'request_id': 'req-1399bf60-8f9b-471d-85d6-f791c646b76c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:31.637901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-1399bf60-8f9b-471d-85d6-f791c646b76c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7dd4ddd5553949d4ad242cf3020c60cc', 'request_id': 'req-1399bf60-8f9b-471d-85d6-f791c646b76c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:31.653536 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-1399bf60-8f9b-471d-85d6-f791c646b76c admin admin] The token cache is being invalidate because project 7dd4ddd5553949d4ad242cf3020c60cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:31.653976 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-1399bf60-8f9b-471d-85d6-f791c646b76c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:31.654797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-1399bf60-8f9b-471d-85d6-f791c646b76c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:31.657515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1022/2080] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:31 2022] DELETE /identity/v3/projects/7dd4ddd5553949d4ad242cf3020c60cc => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:31.661929 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503d6a41-f239-49a3-b3c2-1255fefc69a6 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:31.662217 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503d6a41-f239-49a3-b3c2-1255fefc69a6 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:31.662493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503d6a41-f239-49a3-b3c2-1255fefc69a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:31.684105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-503d6a41-f239-49a3-b3c2-1255fefc69a6 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:31.685333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-503d6a41-f239-49a3-b3c2-1255fefc69a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:31.746656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1059/2081] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:31.754550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284427f6-3696-42e0-9803-2530cfd28d9a None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:31.754866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284427f6-3696-42e0-9803-2530cfd28d9a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:31.755141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284427f6-3696-42e0-9803-2530cfd28d9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:31.775855 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-284427f6-3696-42e0-9803-2530cfd28d9a None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:31.777064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-284427f6-3696-42e0-9803-2530cfd28d9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:31.785264 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1023/2082] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:31.789406 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503d6a41-f239-49a3-b3c2-1255fefc69a6 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:31.798931 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503d6a41-f239-49a3-b3c2-1255fefc69a6 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:31.803524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db84c7e7-3f7a-4d29-a036-519fc0790793 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:31.804147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84c7e7-3f7a-4d29-a036-519fc0790793 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:31.804427 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84c7e7-3f7a-4d29-a036-519fc0790793 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:31.804703 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84c7e7-3f7a-4d29-a036-519fc0790793 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:31.805968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db84c7e7-3f7a-4d29-a036-519fc0790793 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:02:31.806439 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db84c7e7-3f7a-4d29-a036-519fc0790793 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:31.808077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db84c7e7-3f7a-4d29-a036-519fc0790793 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:31.820026 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1060/2083] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:02:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:02:31.825211 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284427f6-3696-42e0-9803-2530cfd28d9a None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:31.834350 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284427f6-3696-42e0-9803-2530cfd28d9a None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:31.839035 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a35eaa-2990-4ac3-9796-ee6fb44c9d0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:31.839676 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a35eaa-2990-4ac3-9796-ee6fb44c9d0f admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:31.840115 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a35eaa-2990-4ac3-9796-ee6fb44c9d0f admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:31.840379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a35eaa-2990-4ac3-9796-ee6fb44c9d0f admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:31.841006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a35eaa-2990-4ac3-9796-ee6fb44c9d0f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:31.842880 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a35eaa-2990-4ac3-9796-ee6fb44c9d0f admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:31.857309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:31.871927 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1024/2084] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:31 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:31.876092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db84c7e7-3f7a-4d29-a036-519fc0790793 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:31.886773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db84c7e7-3f7a-4d29-a036-519fc0790793 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:31.950524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c92146a-e683-4df7-988f-8746448626e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:31.951243 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c92146a-e683-4df7-988f-8746448626e7 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:31.951645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c92146a-e683-4df7-988f-8746448626e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:31.951972 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c92146a-e683-4df7-988f-8746448626e7 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:31.952678 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c92146a-e683-4df7-988f-8746448626e7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:31.954922 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c92146a-e683-4df7-988f-8746448626e7 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:31.979404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1061/2085] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:31 2022] POST /identity/v3/users => generated 293 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:31.983570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a35eaa-2990-4ac3-9796-ee6fb44c9d0f admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:31.997754 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a35eaa-2990-4ac3-9796-ee6fb44c9d0f admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:32.003742 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32729c60-4b61-4eae-ac83-0ffd64c1d343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:32.006285 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32729c60-4b61-4eae-ac83-0ffd64c1d343 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:32.006705 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32729c60-4b61-4eae-ac83-0ffd64c1d343 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:32.007049 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32729c60-4b61-4eae-ac83-0ffd64c1d343 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:32.008113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32729c60-4b61-4eae-ac83-0ffd64c1d343 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:32.009935 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32729c60-4b61-4eae-ac83-0ffd64c1d343 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:32.024138 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1025/2086] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:31 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:32.025304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c92146a-e683-4df7-988f-8746448626e7 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:32.038649 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c92146a-e683-4df7-988f-8746448626e7 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:32.044124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2f0255-d42c-4996-9435-0ac4631fa8ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:32.044739 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2f0255-d42c-4996-9435-0ac4631fa8ef admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:32.045020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2f0255-d42c-4996-9435-0ac4631fa8ef admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:32.045299 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2f0255-d42c-4996-9435-0ac4631fa8ef admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:32.045945 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2f0255-d42c-4996-9435-0ac4631fa8ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:32.047580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2f0255-d42c-4996-9435-0ac4631fa8ef admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:32.064776 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1062/2087] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:32 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:32.066188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32729c60-4b61-4eae-ac83-0ffd64c1d343 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:32.077623 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32729c60-4b61-4eae-ac83-0ffd64c1d343 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:32.082458 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753de35f-9b8e-43ef-82c6-0801cc59aa47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:32.083322 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753de35f-9b8e-43ef-82c6-0801cc59aa47 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:32.083627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753de35f-9b8e-43ef-82c6-0801cc59aa47 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:32.083919 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753de35f-9b8e-43ef-82c6-0801cc59aa47 admin admin] PATH_INFO: `/v3/projects/b1bcc64fff4e4f98a42741f9340ca1a7/users/5c6b23fa6ac440589251d5d80050f2a4/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:32.098878 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753de35f-9b8e-43ef-82c6-0801cc59aa47 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1bcc64fff4e4f98a42741f9340ca1a7, user_id=5c6b23fa6ac440589251d5d80050f2a4, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:32.101552 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753de35f-9b8e-43ef-82c6-0801cc59aa47 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:32.119593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1026/2088] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:32 2022] PUT /identity/v3/projects/b1bcc64fff4e4f98a42741f9340ca1a7/users/5c6b23fa6ac440589251d5d80050f2a4/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:32.130148 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2cbcd9-9cdb-4283-bc21-111c0989b90d None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:32.130366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2cbcd9-9cdb-4283-bc21-111c0989b90d None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:32.130618 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2cbcd9-9cdb-4283-bc21-111c0989b90d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:32.165637 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-2a2cbcd9-9cdb-4283-bc21-111c0989b90d None None] MFA Rules not processed for user `5c6b23fa6ac440589251d5d80050f2a4`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:32.166874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2a2cbcd9-9cdb-4283-bc21-111c0989b90d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:32.231871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753de35f-9b8e-43ef-82c6-0801cc59aa47 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:32.233591 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-753de35f-9b8e-43ef-82c6-0801cc59aa47 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:32.244933 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:32.254820 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1063/2089] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:32 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:32.260134 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2cbcd9-9cdb-4283-bc21-111c0989b90d None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:32.297754 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753de35f-9b8e-43ef-82c6-0801cc59aa47 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:32.301966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce4a6c9-cddb-4d62-9aec-33bca890ce0c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:32.302801 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce4a6c9-cddb-4d62-9aec-33bca890ce0c service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:32.303159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce4a6c9-cddb-4d62-9aec-33bca890ce0c service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:32.303438 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce4a6c9-cddb-4d62-9aec-33bca890ce0c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:32.306735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-9ce4a6c9-cddb-4d62-9aec-33bca890ce0c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:32.363408 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2cbcd9-9cdb-4283-bc21-111c0989b90d None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:32.369820 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a24f93f-5254-4856-8e13-58e40c76fa39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:32.370527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a24f93f-5254-4856-8e13-58e40c76fa39 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:32.371099 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a24f93f-5254-4856-8e13-58e40c76fa39 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:32.371451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a24f93f-5254-4856-8e13-58e40c76fa39 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:32.372343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a24f93f-5254-4856-8e13-58e40c76fa39 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:32.374600 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a24f93f-5254-4856-8e13-58e40c76fa39 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:32.404835 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce4a6c9-cddb-4d62-9aec-33bca890ce0c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:32.406648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce4a6c9-cddb-4d62-9aec-33bca890ce0c service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:32.415233 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1064/2090] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:32 2022] POST /identity/v3/projects => generated 387 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:02:32.418899 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1027/2091] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:32.419726 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a24f93f-5254-4856-8e13-58e40c76fa39 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:32.490312 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a24f93f-5254-4856-8e13-58e40c76fa39 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:32.495843 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8019570d-cece-4b9e-8466-699134fd7427 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:32.496561 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8019570d-cece-4b9e-8466-699134fd7427 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:32.496835 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8019570d-cece-4b9e-8466-699134fd7427 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:32.497098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8019570d-cece-4b9e-8466-699134fd7427 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:32.497728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8019570d-cece-4b9e-8466-699134fd7427 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:32.499674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8019570d-cece-4b9e-8466-699134fd7427 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:32.527714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1065/2092] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:32 2022] POST /identity/v3/users => generated 300 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:02:32.529272 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce4a6c9-cddb-4d62-9aec-33bca890ce0c service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:32.541399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce4a6c9-cddb-4d62-9aec-33bca890ce0c service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:32.546357 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f90f23-8ea5-4c77-a4ba-30e63bf65e1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:32.547048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f90f23-8ea5-4c77-a4ba-30e63bf65e1f admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:32.547389 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f90f23-8ea5-4c77-a4ba-30e63bf65e1f admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:32.547762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f90f23-8ea5-4c77-a4ba-30e63bf65e1f admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:32.548463 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f90f23-8ea5-4c77-a4ba-30e63bf65e1f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:32.550103 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f90f23-8ea5-4c77-a4ba-30e63bf65e1f admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:32.564125 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1028/2093] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:32 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:32.565280 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8019570d-cece-4b9e-8466-699134fd7427 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:32.577011 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8019570d-cece-4b9e-8466-699134fd7427 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:32.581673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8920385d-aa0d-4f9f-857d-fcbd75b0cf5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:32.582328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8920385d-aa0d-4f9f-857d-fcbd75b0cf5f admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:32.582607 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8920385d-aa0d-4f9f-857d-fcbd75b0cf5f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:32.582879 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8920385d-aa0d-4f9f-857d-fcbd75b0cf5f admin admin] PATH_INFO: `/v3/projects/ed852110dfdc43128721dba33d46630e/users/26a4c5c0a8b14e1db78cefb22a1ffc66/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:32.598040 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8920385d-aa0d-4f9f-857d-fcbd75b0cf5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=ed852110dfdc43128721dba33d46630e, user_id=26a4c5c0a8b14e1db78cefb22a1ffc66, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:32.600733 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8920385d-aa0d-4f9f-857d-fcbd75b0cf5f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:32.614207 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1066/2094] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:32 2022] PUT /identity/v3/projects/ed852110dfdc43128721dba33d46630e/users/26a4c5c0a8b14e1db78cefb22a1ffc66/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:32.619570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f90f23-8ea5-4c77-a4ba-30e63bf65e1f admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:32.639756 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f90f23-8ea5-4c77-a4ba-30e63bf65e1f admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:32.714374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:32.730210 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37aa658-bc1f-4bc2-aee1-94ab2be49846 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:32.731257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37aa658-bc1f-4bc2-aee1-94ab2be49846 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:32.731747 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37aa658-bc1f-4bc2-aee1-94ab2be49846 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:32.732201 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37aa658-bc1f-4bc2-aee1-94ab2be49846 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:32.732867 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37aa658-bc1f-4bc2-aee1-94ab2be49846 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:32.734494 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37aa658-bc1f-4bc2-aee1-94ab2be49846 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:32.746463 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1029/2095] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:32 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:32.750641 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8920385d-aa0d-4f9f-857d-fcbd75b0cf5f admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:32.763086 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8920385d-aa0d-4f9f-857d-fcbd75b0cf5f admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:32.771260 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad7ace3-8af3-47a0-bcea-9b8d533e5dab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:32.771968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad7ace3-8af3-47a0-bcea-9b8d533e5dab admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:32.772248 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad7ace3-8af3-47a0-bcea-9b8d533e5dab admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:32.772520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad7ace3-8af3-47a0-bcea-9b8d533e5dab admin admin] PATH_INFO: `/v3/projects/ed852110dfdc43128721dba33d46630e/users/26a4c5c0a8b14e1db78cefb22a1ffc66/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:32.774584 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad7ace3-8af3-47a0-bcea-9b8d533e5dab admin admin] RBAC: Authorizing `identity:create_grant(project_id=ed852110dfdc43128721dba33d46630e, user_id=26a4c5c0a8b14e1db78cefb22a1ffc66, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:32.777860 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad7ace3-8af3-47a0-bcea-9b8d533e5dab admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:32.799745 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1067/2096] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:32 2022] PUT /identity/v3/projects/ed852110dfdc43128721dba33d46630e/users/26a4c5c0a8b14e1db78cefb22a1ffc66/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:32.817993 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864c10f7-6c4e-4657-822b-9a8a990191e0 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:32.818853 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864c10f7-6c4e-4657-822b-9a8a990191e0 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:32.819412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864c10f7-6c4e-4657-822b-9a8a990191e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:32.898181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-864c10f7-6c4e-4657-822b-9a8a990191e0 None None] MFA Rules not processed for user `26a4c5c0a8b14e1db78cefb22a1ffc66`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:32.901405 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-864c10f7-6c4e-4657-822b-9a8a990191e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:33.023281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1030/2097] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:32 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:33.036658 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad7ace3-8af3-47a0-bcea-9b8d533e5dab admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:33.038410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7ad7ace3-8af3-47a0-bcea-9b8d533e5dab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:33.114176 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:33.165099 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad7ace3-8af3-47a0-bcea-9b8d533e5dab admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:33.172883 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d687a666-180e-40d4-b8fc-724601c6c296 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:33.174077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d687a666-180e-40d4-b8fc-724601c6c296 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:33.174358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d687a666-180e-40d4-b8fc-724601c6c296 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:33.174637 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d687a666-180e-40d4-b8fc-724601c6c296 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:33.209881 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d687a666-180e-40d4-b8fc-724601c6c296 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:33.212839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d687a666-180e-40d4-b8fc-724601c6c296 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:33.263164 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1068/2098] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:33 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:34.045311 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864c10f7-6c4e-4657-822b-9a8a990191e0 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:34.212894 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864c10f7-6c4e-4657-822b-9a8a990191e0 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:34.222065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f83c380-4e64-4b12-8d1a-ddacd129bdab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:34.223028 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f83c380-4e64-4b12-8d1a-ddacd129bdab admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:34.223401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f83c380-4e64-4b12-8d1a-ddacd129bdab admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:34.223845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f83c380-4e64-4b12-8d1a-ddacd129bdab admin admin] PATH_INFO: `/v3/users/5c6b23fa6ac440589251d5d80050f2a4` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:34.225379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f83c380-4e64-4b12-8d1a-ddacd129bdab admin admin] RBAC: Authorizing `identity:delete_user(user_id=5c6b23fa6ac440589251d5d80050f2a4)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:34.228317 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f83c380-4e64-4b12-8d1a-ddacd129bdab admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:34.291259 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7f83c380-4e64-4b12-8d1a-ddacd129bdab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5c6b23fa6ac440589251d5d80050f2a4'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:34.301132 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7f83c380-4e64-4b12-8d1a-ddacd129bdab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5c6b23fa6ac440589251d5d80050f2a4'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:34.302072 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7f83c380-4e64-4b12-8d1a-ddacd129bdab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5c6b23fa6ac440589251d5d80050f2a4'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:34.302913 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7f83c380-4e64-4b12-8d1a-ddacd129bdab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5c6b23fa6ac440589251d5d80050f2a4'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:34.327567 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7f83c380-4e64-4b12-8d1a-ddacd129bdab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5c6b23fa6ac440589251d5d80050f2a4'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:34.340265 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:34.355178 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1031/2099] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:34 2022] DELETE /identity/v3/users/5c6b23fa6ac440589251d5d80050f2a4 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:34.359888 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1945fd-0277-4d9f-9b41-03e99e93733c None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:34.360437 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1945fd-0277-4d9f-9b41-03e99e93733c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:34.360437 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1945fd-0277-4d9f-9b41-03e99e93733c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:34.407798 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-8a1945fd-0277-4d9f-9b41-03e99e93733c None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:34.409061 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8a1945fd-0277-4d9f-9b41-03e99e93733c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:34.529392 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1069/2100] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:02:34.541171 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f83c380-4e64-4b12-8d1a-ddacd129bdab admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:34.542763 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-7f83c380-4e64-4b12-8d1a-ddacd129bdab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:34.631869 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f83c380-4e64-4b12-8d1a-ddacd129bdab admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:34.638540 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c1d56e-9636-458d-a10b-09305fa612c6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:34.639293 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c1d56e-9636-458d-a10b-09305fa612c6 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:34.639795 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c1d56e-9636-458d-a10b-09305fa612c6 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:34.640290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c1d56e-9636-458d-a10b-09305fa612c6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:34.663215 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c1d56e-9636-458d-a10b-09305fa612c6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:34.667096 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c1d56e-9636-458d-a10b-09305fa612c6 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:34.708671 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1032/2101] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:35.433759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1945fd-0277-4d9f-9b41-03e99e93733c None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:35.435372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8a1945fd-0277-4d9f-9b41-03e99e93733c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:35.455376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1945fd-0277-4d9f-9b41-03e99e93733c None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:35.491770 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:35.493076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:35.493370 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:35.493669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3 admin admin] PATH_INFO: `/v3/projects/b1bcc64fff4e4f98a42741f9340ca1a7` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:35.504557 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b1bcc64fff4e4f98a42741f9340ca1a7)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:35.506491 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:35.702395 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b1bcc64fff4e4f98a42741f9340ca1a7', 'request_id': 'req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:35.738267 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b1bcc64fff4e4f98a42741f9340ca1a7', 'request_id': 'req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:35.768245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b1bcc64fff4e4f98a42741f9340ca1a7', 'request_id': 'req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:35.808323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3 admin admin] The token cache is being invalidate because project b1bcc64fff4e4f98a42741f9340ca1a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:35.808686 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:35.812200 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:35.818384 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1070/2102] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:35 2022] DELETE /identity/v3/projects/b1bcc64fff4e4f98a42741f9340ca1a7 => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:02:35.823647 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c1d56e-9636-458d-a10b-09305fa612c6 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:35.852575 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c1d56e-9636-458d-a10b-09305fa612c6 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:35.978100 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8b54a9-4783-47aa-b46e-90a09ed550a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:35.979288 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8b54a9-4783-47aa-b46e-90a09ed550a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:35.979946 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8b54a9-4783-47aa-b46e-90a09ed550a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:35.980784 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8b54a9-4783-47aa-b46e-90a09ed550a6 admin admin] PATH_INFO: `/v3/users/26a4c5c0a8b14e1db78cefb22a1ffc66` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:35.983261 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8b54a9-4783-47aa-b46e-90a09ed550a6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=26a4c5c0a8b14e1db78cefb22a1ffc66)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:35.988429 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8b54a9-4783-47aa-b46e-90a09ed550a6 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:36.088773 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2d8b54a9-4783-47aa-b46e-90a09ed550a6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26a4c5c0a8b14e1db78cefb22a1ffc66'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:36.098560 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2d8b54a9-4783-47aa-b46e-90a09ed550a6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26a4c5c0a8b14e1db78cefb22a1ffc66'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:36.099705 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2d8b54a9-4783-47aa-b46e-90a09ed550a6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26a4c5c0a8b14e1db78cefb22a1ffc66'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:36.100509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2d8b54a9-4783-47aa-b46e-90a09ed550a6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26a4c5c0a8b14e1db78cefb22a1ffc66'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:36.115537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2d8b54a9-4783-47aa-b46e-90a09ed550a6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26a4c5c0a8b14e1db78cefb22a1ffc66'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:36.151149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1033/2103] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:35 2022] DELETE /identity/v3/users/26a4c5c0a8b14e1db78cefb22a1ffc66 => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:36.578001 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:36.579628 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:36.598619 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab86153-bc50-4ea9-bcd2-0b3f5f7425c3 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:36.668833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924ac93b-0adc-4d6c-8ae1-aaec616b236f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:36.669564 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924ac93b-0adc-4d6c-8ae1-aaec616b236f admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:36.670048 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924ac93b-0adc-4d6c-8ae1-aaec616b236f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:36.670411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924ac93b-0adc-4d6c-8ae1-aaec616b236f admin admin] PATH_INFO: `/v3/projects/ed852110dfdc43128721dba33d46630e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:36.671687 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924ac93b-0adc-4d6c-8ae1-aaec616b236f admin admin] RBAC: Authorizing `identity:delete_project(project_id=ed852110dfdc43128721dba33d46630e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:36.673831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924ac93b-0adc-4d6c-8ae1-aaec616b236f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:36.675624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:36.735269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-924ac93b-0adc-4d6c-8ae1-aaec616b236f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ed852110dfdc43128721dba33d46630e', 'request_id': 'req-924ac93b-0adc-4d6c-8ae1-aaec616b236f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:36.743951 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-924ac93b-0adc-4d6c-8ae1-aaec616b236f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ed852110dfdc43128721dba33d46630e', 'request_id': 'req-924ac93b-0adc-4d6c-8ae1-aaec616b236f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:36.754772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-924ac93b-0adc-4d6c-8ae1-aaec616b236f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ed852110dfdc43128721dba33d46630e', 'request_id': 'req-924ac93b-0adc-4d6c-8ae1-aaec616b236f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:36.767526 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-924ac93b-0adc-4d6c-8ae1-aaec616b236f admin admin] The token cache is being invalidate because project ed852110dfdc43128721dba33d46630e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:36.767871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-924ac93b-0adc-4d6c-8ae1-aaec616b236f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:36.768631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-924ac93b-0adc-4d6c-8ae1-aaec616b236f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:36.770492 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1071/2104] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:36 2022] DELETE /identity/v3/projects/ed852110dfdc43128721dba33d46630e => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:36.775876 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e01354-0000-4aa9-9958-d917f01170c0 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:36.776187 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e01354-0000-4aa9-9958-d917f01170c0 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:36.776430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e01354-0000-4aa9-9958-d917f01170c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:36.797969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-a1e01354-0000-4aa9-9958-d917f01170c0 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:36.799201 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a1e01354-0000-4aa9-9958-d917f01170c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:36.815159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:36.884426 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1034/2105] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:36.889812 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5282a5c1-b9e9-41e0-81c4-79f217994dc9 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:36.890175 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5282a5c1-b9e9-41e0-81c4-79f217994dc9 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:36.890377 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5282a5c1-b9e9-41e0-81c4-79f217994dc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:36.911305 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-5282a5c1-b9e9-41e0-81c4-79f217994dc9 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:36.912677 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5282a5c1-b9e9-41e0-81c4-79f217994dc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:36.921526 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1072/2106] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:36.925774 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e01354-0000-4aa9-9958-d917f01170c0 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:36.937423 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e01354-0000-4aa9-9958-d917f01170c0 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:36.943117 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283c98b2-8652-4b9e-88f1-ceff5b156c6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:36.943859 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283c98b2-8652-4b9e-88f1-ceff5b156c6d admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:36.944259 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283c98b2-8652-4b9e-88f1-ceff5b156c6d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:36.944714 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283c98b2-8652-4b9e-88f1-ceff5b156c6d admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:36.945470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283c98b2-8652-4b9e-88f1-ceff5b156c6d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:02:36.945977 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283c98b2-8652-4b9e-88f1-ceff5b156c6d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:36.947996 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283c98b2-8652-4b9e-88f1-ceff5b156c6d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:36.961861 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1035/2107] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:02:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:02:36.966120 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5282a5c1-b9e9-41e0-81c4-79f217994dc9 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:36.977262 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5282a5c1-b9e9-41e0-81c4-79f217994dc9 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:36.982706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6297bd-ff25-41a6-b0a8-62291a88c88c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:36.983050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6297bd-ff25-41a6-b0a8-62291a88c88c admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:36.983327 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6297bd-ff25-41a6-b0a8-62291a88c88c admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:36.983631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6297bd-ff25-41a6-b0a8-62291a88c88c admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:36.984279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6297bd-ff25-41a6-b0a8-62291a88c88c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:36.986313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6297bd-ff25-41a6-b0a8-62291a88c88c admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:37.020073 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1073/2108] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:36 2022] POST /identity/v3/projects => generated 383 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:37.024242 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283c98b2-8652-4b9e-88f1-ceff5b156c6d admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:37.035784 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283c98b2-8652-4b9e-88f1-ceff5b156c6d admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:37.128485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46fc909-f338-4f10-ac3a-1e65413bd7c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:37.129197 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46fc909-f338-4f10-ac3a-1e65413bd7c5 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:37.129567 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46fc909-f338-4f10-ac3a-1e65413bd7c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:37.129901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46fc909-f338-4f10-ac3a-1e65413bd7c5 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:37.130596 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46fc909-f338-4f10-ac3a-1e65413bd7c5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:37.133262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46fc909-f338-4f10-ac3a-1e65413bd7c5 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:37.155860 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1036/2109] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:37 2022] POST /identity/v3/users => generated 292 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:37.159905 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6297bd-ff25-41a6-b0a8-62291a88c88c admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:37.171335 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6297bd-ff25-41a6-b0a8-62291a88c88c admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:37.180113 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430a5652-b013-4933-a3a7-4922ea115774 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:37.180794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430a5652-b013-4933-a3a7-4922ea115774 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:37.181128 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430a5652-b013-4933-a3a7-4922ea115774 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:37.181387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430a5652-b013-4933-a3a7-4922ea115774 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:37.181990 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430a5652-b013-4933-a3a7-4922ea115774 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:37.183846 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430a5652-b013-4933-a3a7-4922ea115774 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:37.195818 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1074/2110] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:37 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:37.200257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46fc909-f338-4f10-ac3a-1e65413bd7c5 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:37.210786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46fc909-f338-4f10-ac3a-1e65413bd7c5 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:37.216079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d2bd1c-cf6c-461b-a251-8a2f573047f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:37.216681 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d2bd1c-cf6c-461b-a251-8a2f573047f7 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:37.216958 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d2bd1c-cf6c-461b-a251-8a2f573047f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:37.217225 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d2bd1c-cf6c-461b-a251-8a2f573047f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:37.217864 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d2bd1c-cf6c-461b-a251-8a2f573047f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:37.219442 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d2bd1c-cf6c-461b-a251-8a2f573047f7 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:37.232500 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1037/2111] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:37 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:02:37.236671 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430a5652-b013-4933-a3a7-4922ea115774 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:37.247622 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430a5652-b013-4933-a3a7-4922ea115774 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:37.252475 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e998617-9507-405e-8573-9ded29ef2367 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:37.253093 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e998617-9507-405e-8573-9ded29ef2367 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:37.253375 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e998617-9507-405e-8573-9ded29ef2367 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:37.253653 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e998617-9507-405e-8573-9ded29ef2367 admin admin] PATH_INFO: `/v3/projects/925e2770e2c54869bd5fcef6c2a42b4e/users/289a72d084814dcc9782f56046fb00a3/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:37.276030 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e998617-9507-405e-8573-9ded29ef2367 admin admin] RBAC: Authorizing `identity:create_grant(project_id=925e2770e2c54869bd5fcef6c2a42b4e, user_id=289a72d084814dcc9782f56046fb00a3, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:37.289885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e998617-9507-405e-8573-9ded29ef2367 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:37.317457 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1075/2112] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:37 2022] PUT /identity/v3/projects/925e2770e2c54869bd5fcef6c2a42b4e/users/289a72d084814dcc9782f56046fb00a3/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:37.319912 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed40442e-7747-43d0-92f9-f08d3a72109f None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:37.320250 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed40442e-7747-43d0-92f9-f08d3a72109f None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:37.320601 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed40442e-7747-43d0-92f9-f08d3a72109f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:37.357749 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e998617-9507-405e-8573-9ded29ef2367 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:37.359175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-ed40442e-7747-43d0-92f9-f08d3a72109f None None] MFA Rules not processed for user `289a72d084814dcc9782f56046fb00a3`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:37.360402 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ed40442e-7747-43d0-92f9-f08d3a72109f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:37.360535 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8e998617-9507-405e-8573-9ded29ef2367 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:37.449326 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1038/2113] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:37 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:37.453381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed40442e-7747-43d0-92f9-f08d3a72109f None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:37.454536 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e998617-9507-405e-8573-9ded29ef2367 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:37.459253 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d4ebfa-256e-49ee-ac32-78c0dd5a37b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:37.459925 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d4ebfa-256e-49ee-ac32-78c0dd5a37b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:37.460205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d4ebfa-256e-49ee-ac32-78c0dd5a37b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:37.460597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d4ebfa-256e-49ee-ac32-78c0dd5a37b5 admin admin] PATH_INFO: `/v3/users/75d1901be8064ff5bf79f0268d589e10` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:37.462343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d4ebfa-256e-49ee-ac32-78c0dd5a37b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=75d1901be8064ff5bf79f0268d589e10)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:37.464353 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d4ebfa-256e-49ee-ac32-78c0dd5a37b5 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:37.465563 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed40442e-7747-43d0-92f9-f08d3a72109f None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:37.470479 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90129345-43a6-41a1-9480-3233649415a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:37.471196 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90129345-43a6-41a1-9480-3233649415a6 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:37.471476 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90129345-43a6-41a1-9480-3233649415a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:37.475745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90129345-43a6-41a1-9480-3233649415a6 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:37.475745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90129345-43a6-41a1-9480-3233649415a6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:37.476078 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90129345-43a6-41a1-9480-3233649415a6 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:37.494692 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:37.503631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:37.513345 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1039/2114] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:37 2022] POST /identity/v3/projects => generated 385 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:02:37.517376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90129345-43a6-41a1-9480-3233649415a6 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:37.550611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-23d4ebfa-256e-49ee-ac32-78c0dd5a37b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75d1901be8064ff5bf79f0268d589e10'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:37.551313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-23d4ebfa-256e-49ee-ac32-78c0dd5a37b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75d1901be8064ff5bf79f0268d589e10'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:37.565586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-23d4ebfa-256e-49ee-ac32-78c0dd5a37b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75d1901be8064ff5bf79f0268d589e10'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:37.574197 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-23d4ebfa-256e-49ee-ac32-78c0dd5a37b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75d1901be8064ff5bf79f0268d589e10'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:37.584657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-23d4ebfa-256e-49ee-ac32-78c0dd5a37b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75d1901be8064ff5bf79f0268d589e10'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:37.607847 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1076/2115] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:37 2022] DELETE /identity/v3/users/75d1901be8064ff5bf79f0268d589e10 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:37.624070 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90129345-43a6-41a1-9480-3233649415a6 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:37.686010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd6174d-d78b-4ee7-a596-78db6985cb2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:37.686619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd6174d-d78b-4ee7-a596-78db6985cb2d admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:37.686896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd6174d-d78b-4ee7-a596-78db6985cb2d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:37.687163 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd6174d-d78b-4ee7-a596-78db6985cb2d admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:37.687892 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd6174d-d78b-4ee7-a596-78db6985cb2d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:37.689910 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd6174d-d78b-4ee7-a596-78db6985cb2d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:37.716678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1040/2116] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:37 2022] POST /identity/v3/users => generated 299 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:37.718354 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d4ebfa-256e-49ee-ac32-78c0dd5a37b5 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:37.719847 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-23d4ebfa-256e-49ee-ac32-78c0dd5a37b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:37.732927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d4ebfa-256e-49ee-ac32-78c0dd5a37b5 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:37.737837 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb85bc7a-aff9-497b-a15d-f4cc7c84f6d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:37.738440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb85bc7a-aff9-497b-a15d-f4cc7c84f6d8 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:37.738744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb85bc7a-aff9-497b-a15d-f4cc7c84f6d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:37.739017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb85bc7a-aff9-497b-a15d-f4cc7c84f6d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:37.739829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb85bc7a-aff9-497b-a15d-f4cc7c84f6d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:37.741441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb85bc7a-aff9-497b-a15d-f4cc7c84f6d8 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:37.752726 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1077/2117] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:37 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:37.758375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd6174d-d78b-4ee7-a596-78db6985cb2d admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:37.770765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd6174d-d78b-4ee7-a596-78db6985cb2d admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:37.776808 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439fb82d-1330-4bd5-bf74-c30b7a521e85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:37.777717 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439fb82d-1330-4bd5-bf74-c30b7a521e85 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:37.778117 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439fb82d-1330-4bd5-bf74-c30b7a521e85 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:37.778465 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439fb82d-1330-4bd5-bf74-c30b7a521e85 admin admin] PATH_INFO: `/v3/projects/38d6b807a4a14dfe8fd40b3f57132dbb/users/7f1740137b614f10acc7eecabc017906/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:37.787705 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb85bc7a-aff9-497b-a15d-f4cc7c84f6d8 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:37.790081 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-eb85bc7a-aff9-497b-a15d-f4cc7c84f6d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:37.797871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439fb82d-1330-4bd5-bf74-c30b7a521e85 admin admin] RBAC: Authorizing `identity:create_grant(project_id=38d6b807a4a14dfe8fd40b3f57132dbb, user_id=7f1740137b614f10acc7eecabc017906, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:37.801620 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439fb82d-1330-4bd5-bf74-c30b7a521e85 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:37.805349 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb85bc7a-aff9-497b-a15d-f4cc7c84f6d8 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:37.811384 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1747a84-e954-4c95-af1f-c26129226703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:37.812162 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1747a84-e954-4c95-af1f-c26129226703 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:37.812509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1747a84-e954-4c95-af1f-c26129226703 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:37.812853 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1747a84-e954-4c95-af1f-c26129226703 admin admin] PATH_INFO: `/v3/projects/fdab2d4f39c64f239ded71cc9773c8bb` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:37.814110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1747a84-e954-4c95-af1f-c26129226703 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fdab2d4f39c64f239ded71cc9773c8bb)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:37.816120 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1747a84-e954-4c95-af1f-c26129226703 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:37.819597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1041/2118] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:37 2022] PUT /identity/v3/projects/38d6b807a4a14dfe8fd40b3f57132dbb/users/7f1740137b614f10acc7eecabc017906/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:37.822720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439fb82d-1330-4bd5-bf74-c30b7a521e85 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:37.877582 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b1747a84-e954-4c95-af1f-c26129226703 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fdab2d4f39c64f239ded71cc9773c8bb', 'request_id': 'req-b1747a84-e954-4c95-af1f-c26129226703'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:37.884999 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b1747a84-e954-4c95-af1f-c26129226703 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fdab2d4f39c64f239ded71cc9773c8bb', 'request_id': 'req-b1747a84-e954-4c95-af1f-c26129226703'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:37.892374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b1747a84-e954-4c95-af1f-c26129226703 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fdab2d4f39c64f239ded71cc9773c8bb', 'request_id': 'req-b1747a84-e954-4c95-af1f-c26129226703'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:37.901464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439fb82d-1330-4bd5-bf74-c30b7a521e85 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:37.907075 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b1747a84-e954-4c95-af1f-c26129226703 admin admin] The token cache is being invalidate because project fdab2d4f39c64f239ded71cc9773c8bb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:37.907374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b1747a84-e954-4c95-af1f-c26129226703 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:37.908509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b1747a84-e954-4c95-af1f-c26129226703 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:37.913925 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1078/2119] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:37 2022] DELETE /identity/v3/projects/fdab2d4f39c64f239ded71cc9773c8bb => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:37.914539 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:37.915923 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d774000f-24d7-4684-a98f-785e1fc80cc2 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:37.917927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d774000f-24d7-4684-a98f-785e1fc80cc2 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:37.918198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d774000f-24d7-4684-a98f-785e1fc80cc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:37.942219 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-d774000f-24d7-4684-a98f-785e1fc80cc2 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:37.943748 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d774000f-24d7-4684-a98f-785e1fc80cc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:37.975446 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8536bcd9-c178-4917-ae99-9dfe897e08b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:37.977034 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8536bcd9-c178-4917-ae99-9dfe897e08b7 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:37.977646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8536bcd9-c178-4917-ae99-9dfe897e08b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:37.977933 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8536bcd9-c178-4917-ae99-9dfe897e08b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:37.978587 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8536bcd9-c178-4917-ae99-9dfe897e08b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:37.981928 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8536bcd9-c178-4917-ae99-9dfe897e08b7 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:37.993817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1042/2120] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:37 2022] GET /identity/v3/roles => generated 1420 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:37.998003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8536bcd9-c178-4917-ae99-9dfe897e08b7 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:37.999947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-8536bcd9-c178-4917-ae99-9dfe897e08b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:38.014285 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8536bcd9-c178-4917-ae99-9dfe897e08b7 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:38.019264 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc05e72-f894-4369-82b6-d8b2be145a41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:38.019778 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc05e72-f894-4369-82b6-d8b2be145a41 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:38.020058 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc05e72-f894-4369-82b6-d8b2be145a41 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:38.026984 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc05e72-f894-4369-82b6-d8b2be145a41 admin admin] PATH_INFO: `/v3/projects/38d6b807a4a14dfe8fd40b3f57132dbb/users/7f1740137b614f10acc7eecabc017906/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:38.027168 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1079/2121] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:02:38.027268 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc05e72-f894-4369-82b6-d8b2be145a41 admin admin] RBAC: Authorizing `identity:create_grant(project_id=38d6b807a4a14dfe8fd40b3f57132dbb, user_id=7f1740137b614f10acc7eecabc017906, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:38.027268 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc05e72-f894-4369-82b6-d8b2be145a41 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:38.029086 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91ce157-6646-4fc0-8036-c31bab785856 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:38.029584 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91ce157-6646-4fc0-8036-c31bab785856 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:38.029729 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91ce157-6646-4fc0-8036-c31bab785856 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:38.037513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1043/2122] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:37 2022] PUT /identity/v3/projects/38d6b807a4a14dfe8fd40b3f57132dbb/users/7f1740137b614f10acc7eecabc017906/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:02:38.048159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a0e026-1a8f-4675-b077-b44880e5a18a None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:38.048443 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a0e026-1a8f-4675-b077-b44880e5a18a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:38.048743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a0e026-1a8f-4675-b077-b44880e5a18a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:38.051592 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-a91ce157-6646-4fc0-8036-c31bab785856 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:38.052892 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a91ce157-6646-4fc0-8036-c31bab785856 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:38.066056 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:38.078889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-b1a0e026-1a8f-4675-b077-b44880e5a18a None None] MFA Rules not processed for user `7f1740137b614f10acc7eecabc017906`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:38.080117 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-b1a0e026-1a8f-4675-b077-b44880e5a18a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:38.119579 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1080/2123] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:02:38.123913 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91ce157-6646-4fc0-8036-c31bab785856 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:38.133671 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91ce157-6646-4fc0-8036-c31bab785856 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:38.138521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59587446-4f7d-48c5-ae16-391c4dbba771 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:38.139116 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59587446-4f7d-48c5-ae16-391c4dbba771 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:38.139397 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59587446-4f7d-48c5-ae16-391c4dbba771 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:38.139804 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59587446-4f7d-48c5-ae16-391c4dbba771 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:38.140471 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59587446-4f7d-48c5-ae16-391c4dbba771 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:02:38.140771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59587446-4f7d-48c5-ae16-391c4dbba771 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:38.142367 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59587446-4f7d-48c5-ae16-391c4dbba771 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:38.151592 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1044/2124] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:38 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:02:38.155247 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1081/2125] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:02:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 15:02:38.159314 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a0e026-1a8f-4675-b077-b44880e5a18a None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:38.162525 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59587446-4f7d-48c5-ae16-391c4dbba771 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:38.164481 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-59587446-4f7d-48c5-ae16-391c4dbba771 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:38.170208 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a0e026-1a8f-4675-b077-b44880e5a18a None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:38.174889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f56800-c515-4515-bd9a-ecaa58667ae8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:38.175492 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f56800-c515-4515-bd9a-ecaa58667ae8 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:38.175804 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f56800-c515-4515-bd9a-ecaa58667ae8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:38.176188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f56800-c515-4515-bd9a-ecaa58667ae8 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:38.176841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f56800-c515-4515-bd9a-ecaa58667ae8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:38.178791 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f56800-c515-4515-bd9a-ecaa58667ae8 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:38.208783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1045/2126] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:38 2022] POST /identity/v3/projects => generated 391 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:02:38.213683 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f56800-c515-4515-bd9a-ecaa58667ae8 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:38.229120 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59587446-4f7d-48c5-ae16-391c4dbba771 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:38.285943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f56800-c515-4515-bd9a-ecaa58667ae8 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:38.289170 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcaf6ed-4145-43b2-ba23-d70bf07b0158 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:38.289863 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcaf6ed-4145-43b2-ba23-d70bf07b0158 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:38.290144 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcaf6ed-4145-43b2-ba23-d70bf07b0158 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:38.290418 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcaf6ed-4145-43b2-ba23-d70bf07b0158 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:38.291148 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2eb317-9c63-4209-9da9-bb6492088192 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:38.292099 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2eb317-9c63-4209-9da9-bb6492088192 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:38.292541 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2eb317-9c63-4209-9da9-bb6492088192 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:38.292679 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-abcaf6ed-4145-43b2-ba23-d70bf07b0158 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:38.293094 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2eb317-9c63-4209-9da9-bb6492088192 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:38.294090 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2eb317-9c63-4209-9da9-bb6492088192 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:38.296263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2eb317-9c63-4209-9da9-bb6492088192 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:38.305915 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:38.322613 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1046/2127] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:38 2022] POST /identity/v3/users => generated 296 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:02:38.326295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2eb317-9c63-4209-9da9-bb6492088192 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:38.337746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2eb317-9c63-4209-9da9-bb6492088192 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:38.342529 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bdb4f3-6408-47a4-8dfe-78f7597797ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:38.343155 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bdb4f3-6408-47a4-8dfe-78f7597797ca admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:38.343445 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bdb4f3-6408-47a4-8dfe-78f7597797ca admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:38.344103 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bdb4f3-6408-47a4-8dfe-78f7597797ca admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:38.344416 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bdb4f3-6408-47a4-8dfe-78f7597797ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:38.345987 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bdb4f3-6408-47a4-8dfe-78f7597797ca admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:38.358010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1047/2128] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:38 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:02:38.360557 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bdb4f3-6408-47a4-8dfe-78f7597797ca admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:38.365118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abcaf6ed-4145-43b2-ba23-d70bf07b0158 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:38.366549 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abcaf6ed-4145-43b2-ba23-d70bf07b0158 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:38.370803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bdb4f3-6408-47a4-8dfe-78f7597797ca admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:38.375718 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e71311-ee58-4665-874f-1dafe75dddb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:38.376261 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1082/2129] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:38 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 15:02:38.376392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e71311-ee58-4665-874f-1dafe75dddb7 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:38.376674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e71311-ee58-4665-874f-1dafe75dddb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:38.376947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e71311-ee58-4665-874f-1dafe75dddb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:38.377578 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e71311-ee58-4665-874f-1dafe75dddb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:38.379172 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e71311-ee58-4665-874f-1dafe75dddb7 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:38.391594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1048/2130] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:38 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:02:38.394915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcaf6ed-4145-43b2-ba23-d70bf07b0158 service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:38.405682 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcaf6ed-4145-43b2-ba23-d70bf07b0158 service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:38.410248 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30c84df-ea86-4604-8088-dc747c070e16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:38.410887 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30c84df-ea86-4604-8088-dc747c070e16 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:38.411167 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30c84df-ea86-4604-8088-dc747c070e16 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:38.411441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30c84df-ea86-4604-8088-dc747c070e16 admin admin] PATH_INFO: `/v3/projects/a07f0478d5fc4243aac5c80849e9adc7/users/af2d85d214404e63b10c0473b052b216/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:38.424634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:38.427831 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30c84df-ea86-4604-8088-dc747c070e16 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a07f0478d5fc4243aac5c80849e9adc7, user_id=af2d85d214404e63b10c0473b052b216, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:38.431663 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30c84df-ea86-4604-8088-dc747c070e16 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:38.447134 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1083/2131] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:38 2022] PUT /identity/v3/projects/a07f0478d5fc4243aac5c80849e9adc7/users/af2d85d214404e63b10c0473b052b216/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:38.458878 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b071ec-9a41-480c-993a-2a84cc43069f None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:38.459170 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b071ec-9a41-480c-993a-2a84cc43069f None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:38.459442 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b071ec-9a41-480c-993a-2a84cc43069f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:38.487626 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-38b071ec-9a41-480c-993a-2a84cc43069f None None] MFA Rules not processed for user `af2d85d214404e63b10c0473b052b216`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:38.488863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-38b071ec-9a41-480c-993a-2a84cc43069f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:38.558930 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1049/2132] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:38 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:38.563665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30c84df-ea86-4604-8088-dc747c070e16 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:38.645091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30c84df-ea86-4604-8088-dc747c070e16 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:38.650887 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424b2175-9ff7-4958-ae06-f00320141eb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:38.651672 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424b2175-9ff7-4958-ae06-f00320141eb3 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:38.652065 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424b2175-9ff7-4958-ae06-f00320141eb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:38.652451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424b2175-9ff7-4958-ae06-f00320141eb3 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:38.653366 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424b2175-9ff7-4958-ae06-f00320141eb3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:38.655455 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424b2175-9ff7-4958-ae06-f00320141eb3 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:38.687344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1084/2133] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:38 2022] POST /identity/v3/projects => generated 393 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:38.691441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b071ec-9a41-480c-993a-2a84cc43069f None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:38.706632 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b071ec-9a41-480c-993a-2a84cc43069f None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:38.789863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53c299f-fd6e-47e1-b46d-03f01612c957 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:38.790479 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53c299f-fd6e-47e1-b46d-03f01612c957 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:38.790856 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53c299f-fd6e-47e1-b46d-03f01612c957 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:38.791217 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53c299f-fd6e-47e1-b46d-03f01612c957 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:38.791992 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53c299f-fd6e-47e1-b46d-03f01612c957 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:38.794396 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53c299f-fd6e-47e1-b46d-03f01612c957 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:38.819744 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1050/2134] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:38 2022] POST /identity/v3/users => generated 303 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:38.824022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424b2175-9ff7-4958-ae06-f00320141eb3 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:38.834568 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424b2175-9ff7-4958-ae06-f00320141eb3 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:38.839911 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c96c38-b603-4e59-893b-167c93c35af2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:38.840513 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c96c38-b603-4e59-893b-167c93c35af2 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:38.840795 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c96c38-b603-4e59-893b-167c93c35af2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:38.841103 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c96c38-b603-4e59-893b-167c93c35af2 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:38.841767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c96c38-b603-4e59-893b-167c93c35af2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:38.843352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c96c38-b603-4e59-893b-167c93c35af2 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:38.854580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1085/2135] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:38 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:02:38.858863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53c299f-fd6e-47e1-b46d-03f01612c957 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:38.869533 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53c299f-fd6e-47e1-b46d-03f01612c957 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:38.874185 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda492af-413d-4fe9-88b2-23d195951630 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:38.874828 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda492af-413d-4fe9-88b2-23d195951630 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:38.875109 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda492af-413d-4fe9-88b2-23d195951630 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:38.875383 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda492af-413d-4fe9-88b2-23d195951630 admin admin] PATH_INFO: `/v3/projects/680944c4ad314b698e2d1f9664f14dd0/users/100d7d563bb3430683010f935ae55683/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:38.890105 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda492af-413d-4fe9-88b2-23d195951630 admin admin] RBAC: Authorizing `identity:create_grant(project_id=680944c4ad314b698e2d1f9664f14dd0, user_id=100d7d563bb3430683010f935ae55683, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:38.892795 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda492af-413d-4fe9-88b2-23d195951630 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:38.905691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1051/2136] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:38 2022] PUT /identity/v3/projects/680944c4ad314b698e2d1f9664f14dd0/users/100d7d563bb3430683010f935ae55683/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:38.909807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c96c38-b603-4e59-893b-167c93c35af2 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:38.920458 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c96c38-b603-4e59-893b-167c93c35af2 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:38.976033 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b430c2-371f-4f8f-8a9e-4d7ac2216647 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:38.977125 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b430c2-371f-4f8f-8a9e-4d7ac2216647 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:38.977398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b430c2-371f-4f8f-8a9e-4d7ac2216647 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:38.977663 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b430c2-371f-4f8f-8a9e-4d7ac2216647 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:38.978292 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83b430c2-371f-4f8f-8a9e-4d7ac2216647 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:38.979989 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83b430c2-371f-4f8f-8a9e-4d7ac2216647 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:38.989544 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1086/2137] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:38 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:38.994317 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda492af-413d-4fe9-88b2-23d195951630 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:39.006412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda492af-413d-4fe9-88b2-23d195951630 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:39.011062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dbb96a-22a7-4be7-98ca-6f3cd38c3b0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:39.011737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dbb96a-22a7-4be7-98ca-6f3cd38c3b0d admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:39.012016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dbb96a-22a7-4be7-98ca-6f3cd38c3b0d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:39.012294 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dbb96a-22a7-4be7-98ca-6f3cd38c3b0d admin admin] PATH_INFO: `/v3/projects/680944c4ad314b698e2d1f9664f14dd0/users/100d7d563bb3430683010f935ae55683/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:39.014260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dbb96a-22a7-4be7-98ca-6f3cd38c3b0d admin admin] RBAC: Authorizing `identity:create_grant(project_id=680944c4ad314b698e2d1f9664f14dd0, user_id=100d7d563bb3430683010f935ae55683, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:39.017099 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dbb96a-22a7-4be7-98ca-6f3cd38c3b0d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:39.028689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1052/2138] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:38 2022] PUT /identity/v3/projects/680944c4ad314b698e2d1f9664f14dd0/users/100d7d563bb3430683010f935ae55683/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:39.040172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6841b2e8-a978-4b54-bfaf-86ecc4318e58 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:39.040394 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6841b2e8-a978-4b54-bfaf-86ecc4318e58 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:39.040670 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6841b2e8-a978-4b54-bfaf-86ecc4318e58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:39.068974 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-6841b2e8-a978-4b54-bfaf-86ecc4318e58 None None] MFA Rules not processed for user `100d7d563bb3430683010f935ae55683`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:39.070245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-6841b2e8-a978-4b54-bfaf-86ecc4318e58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:39.117133 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:39.137800 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1087/2139] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:39 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:39.149484 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dbb96a-22a7-4be7-98ca-6f3cd38c3b0d admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:39.151172 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-16dbb96a-22a7-4be7-98ca-6f3cd38c3b0d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:39.161706 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:INSERT=8,keystone:SELECT=92 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:39.208572 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dbb96a-22a7-4be7-98ca-6f3cd38c3b0d admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:39.212912 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5927f5a3-6a67-495e-af30-d39243fac45d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:39.213507 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5927f5a3-6a67-495e-af30-d39243fac45d service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:39.213780 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5927f5a3-6a67-495e-af30-d39243fac45d service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:39.214048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5927f5a3-6a67-495e-af30-d39243fac45d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:39.222808 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5927f5a3-6a67-495e-af30-d39243fac45d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:39.224253 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5927f5a3-6a67-495e-af30-d39243fac45d service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:39.234203 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1053/2140] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:39 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:39.280324 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6841b2e8-a978-4b54-bfaf-86ecc4318e58 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:39.290415 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6841b2e8-a978-4b54-bfaf-86ecc4318e58 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:39.293956 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9caa9479-c4f0-44af-8007-0f3e752a665d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:39.294550 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9caa9479-c4f0-44af-8007-0f3e752a665d service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:39.294828 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9caa9479-c4f0-44af-8007-0f3e752a665d service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:39.295100 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9caa9479-c4f0-44af-8007-0f3e752a665d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:39.354413 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9caa9479-c4f0-44af-8007-0f3e752a665d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:39.355878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9caa9479-c4f0-44af-8007-0f3e752a665d service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:39.364653 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1088/2141] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:39 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:41.322520 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088eb6b8-d5c0-4ee0-baff-cbd43ae81f59 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:41.322520 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088eb6b8-d5c0-4ee0-baff-cbd43ae81f59 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:41.322520 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088eb6b8-d5c0-4ee0-baff-cbd43ae81f59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:41.355870 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-088eb6b8-d5c0-4ee0-baff-cbd43ae81f59 None None] MFA Rules not processed for user `e0d0f8598fac40449bcb2a55ff8df8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:41.357157 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-088eb6b8-d5c0-4ee0-baff-cbd43ae81f59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:41.426580 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1054/2142] 10.222.0.36 () {58 vars in 967 bytes} [Mon Oct 31 15:02:41 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:41.442185 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9caa9479-c4f0-44af-8007-0f3e752a665d service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:41.443745 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-9caa9479-c4f0-44af-8007-0f3e752a665d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:41.543698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9caa9479-c4f0-44af-8007-0f3e752a665d service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:41.550679 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20402da1-1fd1-4fd0-b86d-00a510e31689 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4ae5e20cc5b40ccbdb951f908f5080d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:41.551467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20402da1-1fd1-4fd0-b86d-00a510e31689 service swift] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:41.551760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20402da1-1fd1-4fd0-b86d-00a510e31689 service swift] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:41.552131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20402da1-1fd1-4fd0-b86d-00a510e31689 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:41.563758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20402da1-1fd1-4fd0-b86d-00a510e31689 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:41.565307 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20402da1-1fd1-4fd0-b86d-00a510e31689 service swift] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:41.575356 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1089/2143] 10.222.0.36 () {60 vars in 1384 bytes} [Mon Oct 31 15:02:41 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 31 15:02:42.674881 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088eb6b8-d5c0-4ee0-baff-cbd43ae81f59 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:42.737875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088eb6b8-d5c0-4ee0-baff-cbd43ae81f59 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:42.743253 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8c0d4d-4058-4a71-aa65-4452af75168c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:42.743983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8c0d4d-4058-4a71-aa65-4452af75168c admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:42.744332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8c0d4d-4058-4a71-aa65-4452af75168c admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:42.744672 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8c0d4d-4058-4a71-aa65-4452af75168c admin admin] PATH_INFO: `/v3/users/af2d85d214404e63b10c0473b052b216` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:42.745873 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8c0d4d-4058-4a71-aa65-4452af75168c admin admin] RBAC: Authorizing `identity:delete_user(user_id=af2d85d214404e63b10c0473b052b216)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:42.747805 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8c0d4d-4058-4a71-aa65-4452af75168c admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:42.796075 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20402da1-1fd1-4fd0-b86d-00a510e31689 service swift] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:42.796589 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7c8c0d4d-4058-4a71-aa65-4452af75168c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af2d85d214404e63b10c0473b052b216'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:42.803930 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7c8c0d4d-4058-4a71-aa65-4452af75168c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af2d85d214404e63b10c0473b052b216'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:42.805224 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7c8c0d4d-4058-4a71-aa65-4452af75168c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af2d85d214404e63b10c0473b052b216'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:42.805827 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7c8c0d4d-4058-4a71-aa65-4452af75168c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af2d85d214404e63b10c0473b052b216'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:42.806081 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20402da1-1fd1-4fd0-b86d-00a510e31689 service swift] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:42.808381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:42.810782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b90de0d-c67c-4664-9710-dc409badf75e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:42.811389 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b90de0d-c67c-4664-9710-dc409badf75e service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:42.811697 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b90de0d-c67c-4664-9710-dc409badf75e service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:42.811965 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b90de0d-c67c-4664-9710-dc409badf75e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:42.814018 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2b90de0d-c67c-4664-9710-dc409badf75e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:42.819776 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-7c8c0d4d-4058-4a71-aa65-4452af75168c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af2d85d214404e63b10c0473b052b216'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:42.832892 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1055/2144] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:42 2022] DELETE /identity/v3/users/af2d85d214404e63b10c0473b052b216 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:42.838097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e80c2a4-e764-44c4-b1d5-c30aa316e56f None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:42.838410 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e80c2a4-e764-44c4-b1d5-c30aa316e56f None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:42.838686 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e80c2a4-e764-44c4-b1d5-c30aa316e56f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:42.859668 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-8e80c2a4-e764-44c4-b1d5-c30aa316e56f None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:42.861157 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-8e80c2a4-e764-44c4-b1d5-c30aa316e56f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:42.881967 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b90de0d-c67c-4664-9710-dc409badf75e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:42.883676 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b90de0d-c67c-4664-9710-dc409badf75e service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:42.893398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1090/2145] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:42 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 15:02:42.920122 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1056/2146] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:02:42.930096 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b90de0d-c67c-4664-9710-dc409badf75e service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:42.931568 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2b90de0d-c67c-4664-9710-dc409badf75e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:42.943510 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:42.993555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b90de0d-c67c-4664-9710-dc409badf75e service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:43.043597 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fef297-7af5-481e-a7de-9efe2871386b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:43.044203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fef297-7af5-481e-a7de-9efe2871386b service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:43.044480 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fef297-7af5-481e-a7de-9efe2871386b service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:43.044767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fef297-7af5-481e-a7de-9efe2871386b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:43.054118 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42fef297-7af5-481e-a7de-9efe2871386b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:43.055736 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42fef297-7af5-481e-a7de-9efe2871386b service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:43.067150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1091/2147] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:43.349358 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e80c2a4-e764-44c4-b1d5-c30aa316e56f None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:43.350877 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-8e80c2a4-e764-44c4-b1d5-c30aa316e56f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:43.363270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e80c2a4-e764-44c4-b1d5-c30aa316e56f None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:43.368362 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d3147a-46b8-48a2-8803-387149bcad44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:43.368997 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d3147a-46b8-48a2-8803-387149bcad44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:43.369276 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d3147a-46b8-48a2-8803-387149bcad44 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:43.369676 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d3147a-46b8-48a2-8803-387149bcad44 admin admin] PATH_INFO: `/v3/projects/a07f0478d5fc4243aac5c80849e9adc7` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:43.371215 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d3147a-46b8-48a2-8803-387149bcad44 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a07f0478d5fc4243aac5c80849e9adc7)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:43.372790 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d3147a-46b8-48a2-8803-387149bcad44 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:43.419219 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-14d3147a-46b8-48a2-8803-387149bcad44 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a07f0478d5fc4243aac5c80849e9adc7', 'request_id': 'req-14d3147a-46b8-48a2-8803-387149bcad44'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:43.425777 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-14d3147a-46b8-48a2-8803-387149bcad44 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a07f0478d5fc4243aac5c80849e9adc7', 'request_id': 'req-14d3147a-46b8-48a2-8803-387149bcad44'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:43.431910 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-14d3147a-46b8-48a2-8803-387149bcad44 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a07f0478d5fc4243aac5c80849e9adc7', 'request_id': 'req-14d3147a-46b8-48a2-8803-387149bcad44'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:43.442815 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-14d3147a-46b8-48a2-8803-387149bcad44 admin admin] The token cache is being invalidate because project a07f0478d5fc4243aac5c80849e9adc7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:43.443100 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-14d3147a-46b8-48a2-8803-387149bcad44 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:43.443763 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-14d3147a-46b8-48a2-8803-387149bcad44 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:43.445758 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1057/2148] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:43 2022] DELETE /identity/v3/projects/a07f0478d5fc4243aac5c80849e9adc7 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:43.450108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fef297-7af5-481e-a7de-9efe2871386b service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:43.459523 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fef297-7af5-481e-a7de-9efe2871386b service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:43.517673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc98640-9e5f-4e10-ba63-a444dd82783b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:43.518318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc98640-9e5f-4e10-ba63-a444dd82783b admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:43.518775 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc98640-9e5f-4e10-ba63-a444dd82783b admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:43.519050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc98640-9e5f-4e10-ba63-a444dd82783b admin admin] PATH_INFO: `/v3/users/100d7d563bb3430683010f935ae55683` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:43.520204 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc98640-9e5f-4e10-ba63-a444dd82783b admin admin] RBAC: Authorizing `identity:delete_user(user_id=100d7d563bb3430683010f935ae55683)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:43.522131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc98640-9e5f-4e10-ba63-a444dd82783b admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:43.572031 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5fc98640-9e5f-4e10-ba63-a444dd82783b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '100d7d563bb3430683010f935ae55683'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:43.572616 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5fc98640-9e5f-4e10-ba63-a444dd82783b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '100d7d563bb3430683010f935ae55683'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:43.584559 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5fc98640-9e5f-4e10-ba63-a444dd82783b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '100d7d563bb3430683010f935ae55683'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:43.589617 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:43.593608 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5fc98640-9e5f-4e10-ba63-a444dd82783b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '100d7d563bb3430683010f935ae55683'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:43.601634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-5fc98640-9e5f-4e10-ba63-a444dd82783b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '100d7d563bb3430683010f935ae55683'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:43.607438 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1092/2149] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:43 2022] DELETE /identity/v3/users/100d7d563bb3430683010f935ae55683 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:43.885969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d3147a-46b8-48a2-8803-387149bcad44 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:43.887442 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-14d3147a-46b8-48a2-8803-387149bcad44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:43.901270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d3147a-46b8-48a2-8803-387149bcad44 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:43.958418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccc3f6f-ea25-4099-b6e9-68c0d36784b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:43.959028 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccc3f6f-ea25-4099-b6e9-68c0d36784b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:43.959305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccc3f6f-ea25-4099-b6e9-68c0d36784b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:43.959598 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccc3f6f-ea25-4099-b6e9-68c0d36784b3 admin admin] PATH_INFO: `/v3/projects/680944c4ad314b698e2d1f9664f14dd0` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:43.960819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccc3f6f-ea25-4099-b6e9-68c0d36784b3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=680944c4ad314b698e2d1f9664f14dd0)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:43.962739 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccc3f6f-ea25-4099-b6e9-68c0d36784b3 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:43.964208 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:44.012574 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8ccc3f6f-ea25-4099-b6e9-68c0d36784b3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '680944c4ad314b698e2d1f9664f14dd0', 'request_id': 'req-8ccc3f6f-ea25-4099-b6e9-68c0d36784b3'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:44.019771 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8ccc3f6f-ea25-4099-b6e9-68c0d36784b3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '680944c4ad314b698e2d1f9664f14dd0', 'request_id': 'req-8ccc3f6f-ea25-4099-b6e9-68c0d36784b3'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:44.025350 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8ccc3f6f-ea25-4099-b6e9-68c0d36784b3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '680944c4ad314b698e2d1f9664f14dd0', 'request_id': 'req-8ccc3f6f-ea25-4099-b6e9-68c0d36784b3'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:44.036139 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8ccc3f6f-ea25-4099-b6e9-68c0d36784b3 admin admin] The token cache is being invalidate because project 680944c4ad314b698e2d1f9664f14dd0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:44.036423 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8ccc3f6f-ea25-4099-b6e9-68c0d36784b3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:44.037052 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8ccc3f6f-ea25-4099-b6e9-68c0d36784b3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:44.039446 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1058/2150] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:43 2022] DELETE /identity/v3/projects/680944c4ad314b698e2d1f9664f14dd0 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:44.043878 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e84dcd-485b-46e1-a669-32bd411069c9 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:44.044354 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e84dcd-485b-46e1-a669-32bd411069c9 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:44.044448 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e84dcd-485b-46e1-a669-32bd411069c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:44.064758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-61e84dcd-485b-46e1-a669-32bd411069c9 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:44.065948 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-61e84dcd-485b-46e1-a669-32bd411069c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:44.123951 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1093/2151] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:02:44.131544 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed8c961-4903-4661-9a74-eb1d28949a59 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:44.131844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed8c961-4903-4661-9a74-eb1d28949a59 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:44.132564 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed8c961-4903-4661-9a74-eb1d28949a59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:44.153450 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-1ed8c961-4903-4661-9a74-eb1d28949a59 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:44.154632 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-1ed8c961-4903-4661-9a74-eb1d28949a59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:44.162884 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1059/2152] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:44.167573 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e84dcd-485b-46e1-a669-32bd411069c9 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:44.177297 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e84dcd-485b-46e1-a669-32bd411069c9 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:44.182917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d69728a-6bc9-4731-b88f-8f3bd361beac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:44.183511 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d69728a-6bc9-4731-b88f-8f3bd361beac admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:44.183823 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d69728a-6bc9-4731-b88f-8f3bd361beac admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:44.184099 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d69728a-6bc9-4731-b88f-8f3bd361beac admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:44.184746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d69728a-6bc9-4731-b88f-8f3bd361beac admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:02:44.185050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d69728a-6bc9-4731-b88f-8f3bd361beac admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:44.186643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d69728a-6bc9-4731-b88f-8f3bd361beac admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:44.198932 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1094/2153] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:02:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:02:44.203027 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed8c961-4903-4661-9a74-eb1d28949a59 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:44.212247 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed8c961-4903-4661-9a74-eb1d28949a59 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:44.217023 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a252d0dc-b930-4d40-bddc-5b8fcbbe3c63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:44.217624 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a252d0dc-b930-4d40-bddc-5b8fcbbe3c63 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:44.217902 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a252d0dc-b930-4d40-bddc-5b8fcbbe3c63 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:44.218172 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a252d0dc-b930-4d40-bddc-5b8fcbbe3c63 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:44.218805 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a252d0dc-b930-4d40-bddc-5b8fcbbe3c63 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:44.220811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a252d0dc-b930-4d40-bddc-5b8fcbbe3c63 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:44.263779 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1060/2154] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:44 2022] POST /identity/v3/projects => generated 401 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:44.263894 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d69728a-6bc9-4731-b88f-8f3bd361beac admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:44.266788 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d69728a-6bc9-4731-b88f-8f3bd361beac admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:44.328704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a38b244-7466-47f1-a124-1098c8d4b040 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:44.329848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a38b244-7466-47f1-a124-1098c8d4b040 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:44.330295 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a38b244-7466-47f1-a124-1098c8d4b040 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:44.331064 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a38b244-7466-47f1-a124-1098c8d4b040 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:44.332090 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a38b244-7466-47f1-a124-1098c8d4b040 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:44.334152 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a38b244-7466-47f1-a124-1098c8d4b040 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:44.357187 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1095/2155] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:44 2022] POST /identity/v3/users => generated 301 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:44.361408 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a252d0dc-b930-4d40-bddc-5b8fcbbe3c63 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:44.373114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a252d0dc-b930-4d40-bddc-5b8fcbbe3c63 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:44.377837 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63c58d8-7247-4d91-9fcf-6c5a87bf7fe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:44.378445 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63c58d8-7247-4d91-9fcf-6c5a87bf7fe9 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:44.378793 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63c58d8-7247-4d91-9fcf-6c5a87bf7fe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:44.379131 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63c58d8-7247-4d91-9fcf-6c5a87bf7fe9 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:44.379890 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63c58d8-7247-4d91-9fcf-6c5a87bf7fe9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:44.381676 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63c58d8-7247-4d91-9fcf-6c5a87bf7fe9 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:44.393167 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1061/2156] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:44 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:44.397294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a38b244-7466-47f1-a124-1098c8d4b040 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:44.409528 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a38b244-7466-47f1-a124-1098c8d4b040 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:44.416177 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e533ce2-9070-42f1-bb87-6500eb46420b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:44.416778 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e533ce2-9070-42f1-bb87-6500eb46420b admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:44.419279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e533ce2-9070-42f1-bb87-6500eb46420b admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:44.419279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e533ce2-9070-42f1-bb87-6500eb46420b admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:44.419279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e533ce2-9070-42f1-bb87-6500eb46420b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:44.419929 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e533ce2-9070-42f1-bb87-6500eb46420b admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:44.432094 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1096/2157] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:44 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:44.436354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63c58d8-7247-4d91-9fcf-6c5a87bf7fe9 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:44.447640 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63c58d8-7247-4d91-9fcf-6c5a87bf7fe9 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:44.452433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be81479b-9a22-45cd-a86a-7860ff2dcb17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:44.453204 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be81479b-9a22-45cd-a86a-7860ff2dcb17 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:44.453488 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be81479b-9a22-45cd-a86a-7860ff2dcb17 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:44.453761 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be81479b-9a22-45cd-a86a-7860ff2dcb17 admin admin] PATH_INFO: `/v3/projects/7762aed01fe74afdb22fb68970903942/users/d08c674729f1490597fb54dc197c1d79/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:44.470519 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be81479b-9a22-45cd-a86a-7860ff2dcb17 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7762aed01fe74afdb22fb68970903942, user_id=d08c674729f1490597fb54dc197c1d79, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:44.473234 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be81479b-9a22-45cd-a86a-7860ff2dcb17 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:44.487708 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1062/2158] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:44 2022] PUT /identity/v3/projects/7762aed01fe74afdb22fb68970903942/users/d08c674729f1490597fb54dc197c1d79/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:44.499169 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ca8198-0ef9-4fae-a12d-7e7788ddcb7c None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:44.499453 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ca8198-0ef9-4fae-a12d-7e7788ddcb7c None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:44.499764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ca8198-0ef9-4fae-a12d-7e7788ddcb7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:44.534971 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-28ca8198-0ef9-4fae-a12d-7e7788ddcb7c None None] MFA Rules not processed for user `d08c674729f1490597fb54dc197c1d79`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:44.536295 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-28ca8198-0ef9-4fae-a12d-7e7788ddcb7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:44.581010 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:44.610203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1097/2159] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:44 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:44.614929 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be81479b-9a22-45cd-a86a-7860ff2dcb17 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:44.686341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be81479b-9a22-45cd-a86a-7860ff2dcb17 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:44.691961 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b0972a-4082-4b46-aba1-e4b70dc33382 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:44.692570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b0972a-4082-4b46-aba1-e4b70dc33382 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:44.692852 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b0972a-4082-4b46-aba1-e4b70dc33382 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:44.693126 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b0972a-4082-4b46-aba1-e4b70dc33382 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:44.693761 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b0972a-4082-4b46-aba1-e4b70dc33382 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:44.696346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b0972a-4082-4b46-aba1-e4b70dc33382 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:44.729445 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1063/2160] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:44 2022] POST /identity/v3/projects => generated 401 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:44.733550 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ca8198-0ef9-4fae-a12d-7e7788ddcb7c None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:44.745615 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ca8198-0ef9-4fae-a12d-7e7788ddcb7c None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:44.806040 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40ad15e-fa48-413f-9d5b-5559ac8f9f22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:44.806647 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40ad15e-fa48-413f-9d5b-5559ac8f9f22 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:44.806932 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40ad15e-fa48-413f-9d5b-5559ac8f9f22 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:44.807209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40ad15e-fa48-413f-9d5b-5559ac8f9f22 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:44.807884 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40ad15e-fa48-413f-9d5b-5559ac8f9f22 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:44.809808 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40ad15e-fa48-413f-9d5b-5559ac8f9f22 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:44.831733 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1098/2161] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:44 2022] POST /identity/v3/users => generated 307 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:02:44.836473 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b0972a-4082-4b46-aba1-e4b70dc33382 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:44.846652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b0972a-4082-4b46-aba1-e4b70dc33382 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:44.851570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f69f65-bcf1-4406-b212-b8d464dcc162 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:44.852166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f69f65-bcf1-4406-b212-b8d464dcc162 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:44.852445 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f69f65-bcf1-4406-b212-b8d464dcc162 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:44.852714 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f69f65-bcf1-4406-b212-b8d464dcc162 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:44.853345 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f69f65-bcf1-4406-b212-b8d464dcc162 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:44.854921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f69f65-bcf1-4406-b212-b8d464dcc162 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:44.860062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:44.866870 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1064/2162] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:44.871138 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40ad15e-fa48-413f-9d5b-5559ac8f9f22 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:44.881207 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40ad15e-fa48-413f-9d5b-5559ac8f9f22 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:44.886384 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927d813a-79f6-4358-9928-7449bc947212 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:44.887052 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927d813a-79f6-4358-9928-7449bc947212 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:44.887333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927d813a-79f6-4358-9928-7449bc947212 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:44.887660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927d813a-79f6-4358-9928-7449bc947212 admin admin] PATH_INFO: `/v3/projects/9c7f43c375e34c229ebf8d826d6cd88b/users/45d026015d6149048ffb10b5feef847b/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:44.902810 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927d813a-79f6-4358-9928-7449bc947212 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c7f43c375e34c229ebf8d826d6cd88b, user_id=45d026015d6149048ffb10b5feef847b, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:44.905480 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927d813a-79f6-4358-9928-7449bc947212 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:44.918228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1099/2163] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:44 2022] PUT /identity/v3/projects/9c7f43c375e34c229ebf8d826d6cd88b/users/45d026015d6149048ffb10b5feef847b/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:44.922400 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f69f65-bcf1-4406-b212-b8d464dcc162 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:44.932765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f69f65-bcf1-4406-b212-b8d464dcc162 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:44.989256 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322262b1-4918-42ec-af0d-bf7958231a63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:44.989864 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322262b1-4918-42ec-af0d-bf7958231a63 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:44.990159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322262b1-4918-42ec-af0d-bf7958231a63 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:44.990436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322262b1-4918-42ec-af0d-bf7958231a63 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:44.991081 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322262b1-4918-42ec-af0d-bf7958231a63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:44.992792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322262b1-4918-42ec-af0d-bf7958231a63 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:45.002634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1065/2164] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:44 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:45.006746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927d813a-79f6-4358-9928-7449bc947212 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:45.017975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927d813a-79f6-4358-9928-7449bc947212 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:45.022724 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3a53ee-016e-4074-9e55-aaf19ec085ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:45.023466 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3a53ee-016e-4074-9e55-aaf19ec085ff admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:45.023774 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3a53ee-016e-4074-9e55-aaf19ec085ff admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:45.024049 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3a53ee-016e-4074-9e55-aaf19ec085ff admin admin] PATH_INFO: `/v3/projects/9c7f43c375e34c229ebf8d826d6cd88b/users/45d026015d6149048ffb10b5feef847b/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:45.025941 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3a53ee-016e-4074-9e55-aaf19ec085ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c7f43c375e34c229ebf8d826d6cd88b, user_id=45d026015d6149048ffb10b5feef847b, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:45.028617 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3a53ee-016e-4074-9e55-aaf19ec085ff admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:45.039672 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1100/2165] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:45 2022] PUT /identity/v3/projects/9c7f43c375e34c229ebf8d826d6cd88b/users/45d026015d6149048ffb10b5feef847b/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:45.049841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05c9988-34d6-4451-a730-783039694df4 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:45.050138 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05c9988-34d6-4451-a730-783039694df4 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:45.050418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05c9988-34d6-4451-a730-783039694df4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:45.079272 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-b05c9988-34d6-4451-a730-783039694df4 None None] MFA Rules not processed for user `45d026015d6149048ffb10b5feef847b`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:45.080556 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-b05c9988-34d6-4451-a730-783039694df4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:45.149164 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1066/2166] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:45 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:45.159867 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3a53ee-016e-4074-9e55-aaf19ec085ff admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:45.161652 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2e3a53ee-016e-4074-9e55-aaf19ec085ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:45.218987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3a53ee-016e-4074-9e55-aaf19ec085ff admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:45.222610 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d53daa-9f47-4955-ae45-eeaa81d5539d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:45.223204 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d53daa-9f47-4955-ae45-eeaa81d5539d service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:45.223485 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d53daa-9f47-4955-ae45-eeaa81d5539d service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:45.223787 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d53daa-9f47-4955-ae45-eeaa81d5539d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:45.233526 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d53daa-9f47-4955-ae45-eeaa81d5539d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:45.234925 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d53daa-9f47-4955-ae45-eeaa81d5539d service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:45.244424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1101/2167] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:45 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:45.324753 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05c9988-34d6-4451-a730-783039694df4 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:45.334819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05c9988-34d6-4451-a730-783039694df4 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:45.339187 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8407ed-097f-44d0-a443-9fb08a63166b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:45.339813 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8407ed-097f-44d0-a443-9fb08a63166b service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:45.340097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8407ed-097f-44d0-a443-9fb08a63166b service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:45.340372 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8407ed-097f-44d0-a443-9fb08a63166b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:45.396373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8407ed-097f-44d0-a443-9fb08a63166b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:45.398414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8407ed-097f-44d0-a443-9fb08a63166b service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:45.407382 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1067/2168] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:45 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:47.841643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d53daa-9f47-4955-ae45-eeaa81d5539d service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:47.843049 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a3d53daa-9f47-4955-ae45-eeaa81d5539d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:47.884489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:47.907997 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d53daa-9f47-4955-ae45-eeaa81d5539d service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:47.912549 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49aa8b8-a838-41ca-ac85-a9b58d2551a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:47.913212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49aa8b8-a838-41ca-ac85-a9b58d2551a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:47.913574 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49aa8b8-a838-41ca-ac85-a9b58d2551a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:47.913856 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49aa8b8-a838-41ca-ac85-a9b58d2551a1 admin admin] PATH_INFO: `/v3/users/289a72d084814dcc9782f56046fb00a3` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:47.914971 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49aa8b8-a838-41ca-ac85-a9b58d2551a1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=289a72d084814dcc9782f56046fb00a3)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:47.916912 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49aa8b8-a838-41ca-ac85-a9b58d2551a1 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:47.965001 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e49aa8b8-a838-41ca-ac85-a9b58d2551a1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '289a72d084814dcc9782f56046fb00a3'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:47.965735 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e49aa8b8-a838-41ca-ac85-a9b58d2551a1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '289a72d084814dcc9782f56046fb00a3'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:47.977828 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e49aa8b8-a838-41ca-ac85-a9b58d2551a1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '289a72d084814dcc9782f56046fb00a3'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:47.984703 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e49aa8b8-a838-41ca-ac85-a9b58d2551a1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '289a72d084814dcc9782f56046fb00a3'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:47.992013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e49aa8b8-a838-41ca-ac85-a9b58d2551a1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '289a72d084814dcc9782f56046fb00a3'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:47.997825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1102/2169] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:47 2022] DELETE /identity/v3/users/289a72d084814dcc9782f56046fb00a3 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:48.002366 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b644253-3241-4e8f-8e34-cef2c58fc8a4 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:48.002822 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b644253-3241-4e8f-8e34-cef2c58fc8a4 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:48.002932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b644253-3241-4e8f-8e34-cef2c58fc8a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:48.023206 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-7b644253-3241-4e8f-8e34-cef2c58fc8a4 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:48.024628 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-7b644253-3241-4e8f-8e34-cef2c58fc8a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:48.037016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:48.084140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1068/2170] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:48.094246 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49aa8b8-a838-41ca-ac85-a9b58d2551a1 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:48.095682 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e49aa8b8-a838-41ca-ac85-a9b58d2551a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:48.159690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49aa8b8-a838-41ca-ac85-a9b58d2551a1 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:48.163706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a99049a-4a1d-43d5-a946-397caca9504a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:48.164699 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a99049a-4a1d-43d5-a946-397caca9504a service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:48.165112 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a99049a-4a1d-43d5-a946-397caca9504a service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:48.165552 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a99049a-4a1d-43d5-a946-397caca9504a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:48.179690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a99049a-4a1d-43d5-a946-397caca9504a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:48.181350 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a99049a-4a1d-43d5-a946-397caca9504a service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:48.191013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1103/2171] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:48.475403 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b644253-3241-4e8f-8e34-cef2c58fc8a4 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:48.477723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-7b644253-3241-4e8f-8e34-cef2c58fc8a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:48.492205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b644253-3241-4e8f-8e34-cef2c58fc8a4 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:48.502936 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b1b2c6-b25a-488e-b8b6-3195d119536b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:48.503604 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b1b2c6-b25a-488e-b8b6-3195d119536b admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:48.503889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b1b2c6-b25a-488e-b8b6-3195d119536b admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:48.504299 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b1b2c6-b25a-488e-b8b6-3195d119536b admin admin] PATH_INFO: `/v3/projects/925e2770e2c54869bd5fcef6c2a42b4e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:48.506618 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b1b2c6-b25a-488e-b8b6-3195d119536b admin admin] RBAC: Authorizing `identity:delete_project(project_id=925e2770e2c54869bd5fcef6c2a42b4e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:48.508969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b1b2c6-b25a-488e-b8b6-3195d119536b admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:48.559105 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-18b1b2c6-b25a-488e-b8b6-3195d119536b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '925e2770e2c54869bd5fcef6c2a42b4e', 'request_id': 'req-18b1b2c6-b25a-488e-b8b6-3195d119536b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:48.566761 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-18b1b2c6-b25a-488e-b8b6-3195d119536b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '925e2770e2c54869bd5fcef6c2a42b4e', 'request_id': 'req-18b1b2c6-b25a-488e-b8b6-3195d119536b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:48.573865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-18b1b2c6-b25a-488e-b8b6-3195d119536b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '925e2770e2c54869bd5fcef6c2a42b4e', 'request_id': 'req-18b1b2c6-b25a-488e-b8b6-3195d119536b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:48.585037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-18b1b2c6-b25a-488e-b8b6-3195d119536b admin admin] The token cache is being invalidate because project 925e2770e2c54869bd5fcef6c2a42b4e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:48.585871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-18b1b2c6-b25a-488e-b8b6-3195d119536b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:48.586524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-18b1b2c6-b25a-488e-b8b6-3195d119536b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:48.588067 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1069/2172] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:48 2022] DELETE /identity/v3/projects/925e2770e2c54869bd5fcef6c2a42b4e => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:48.592049 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a99049a-4a1d-43d5-a946-397caca9504a service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:48.602155 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a99049a-4a1d-43d5-a946-397caca9504a service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:48.656684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636f643d-4b00-473c-b5c5-6df5787c747f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:48.657307 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636f643d-4b00-473c-b5c5-6df5787c747f admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:48.657584 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636f643d-4b00-473c-b5c5-6df5787c747f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:48.657856 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636f643d-4b00-473c-b5c5-6df5787c747f admin admin] PATH_INFO: `/v3/users/7f1740137b614f10acc7eecabc017906` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:48.658966 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636f643d-4b00-473c-b5c5-6df5787c747f admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f1740137b614f10acc7eecabc017906)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:48.660902 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636f643d-4b00-473c-b5c5-6df5787c747f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:48.662330 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:48.708187 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-636f643d-4b00-473c-b5c5-6df5787c747f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f1740137b614f10acc7eecabc017906'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:48.708846 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-636f643d-4b00-473c-b5c5-6df5787c747f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f1740137b614f10acc7eecabc017906'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:48.720630 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-636f643d-4b00-473c-b5c5-6df5787c747f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f1740137b614f10acc7eecabc017906'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:48.727972 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-636f643d-4b00-473c-b5c5-6df5787c747f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f1740137b614f10acc7eecabc017906'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:48.735053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-636f643d-4b00-473c-b5c5-6df5787c747f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f1740137b614f10acc7eecabc017906'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:48.740866 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1104/2173] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:48 2022] DELETE /identity/v3/users/7f1740137b614f10acc7eecabc017906 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:49.009965 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b1b2c6-b25a-488e-b8b6-3195d119536b admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:49.011443 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-18b1b2c6-b25a-488e-b8b6-3195d119536b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:49.029954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b1b2c6-b25a-488e-b8b6-3195d119536b admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:49.089237 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08710ac7-0eac-4836-930e-756d9812c261 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:49.089811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08710ac7-0eac-4836-930e-756d9812c261 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:49.090086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08710ac7-0eac-4836-930e-756d9812c261 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:49.090354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08710ac7-0eac-4836-930e-756d9812c261 admin admin] PATH_INFO: `/v3/projects/38d6b807a4a14dfe8fd40b3f57132dbb` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:49.091481 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08710ac7-0eac-4836-930e-756d9812c261 admin admin] RBAC: Authorizing `identity:delete_project(project_id=38d6b807a4a14dfe8fd40b3f57132dbb)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:49.093429 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08710ac7-0eac-4836-930e-756d9812c261 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:49.138521 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:49.141948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-08710ac7-0eac-4836-930e-756d9812c261 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '38d6b807a4a14dfe8fd40b3f57132dbb', 'request_id': 'req-08710ac7-0eac-4836-930e-756d9812c261'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:49.150008 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-08710ac7-0eac-4836-930e-756d9812c261 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '38d6b807a4a14dfe8fd40b3f57132dbb', 'request_id': 'req-08710ac7-0eac-4836-930e-756d9812c261'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:49.155788 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-08710ac7-0eac-4836-930e-756d9812c261 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '38d6b807a4a14dfe8fd40b3f57132dbb', 'request_id': 'req-08710ac7-0eac-4836-930e-756d9812c261'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:49.167689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-08710ac7-0eac-4836-930e-756d9812c261 admin admin] The token cache is being invalidate because project 38d6b807a4a14dfe8fd40b3f57132dbb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:49.167994 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-08710ac7-0eac-4836-930e-756d9812c261 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:49.168648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-08710ac7-0eac-4836-930e-756d9812c261 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:49.170224 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1070/2174] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:49 2022] DELETE /identity/v3/projects/38d6b807a4a14dfe8fd40b3f57132dbb => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:49.176179 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef0d082-6c27-4165-b461-7a4fe936a03d None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:49.176667 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef0d082-6c27-4165-b461-7a4fe936a03d None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:49.176790 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef0d082-6c27-4165-b461-7a4fe936a03d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:49.195939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-1ef0d082-6c27-4165-b461-7a4fe936a03d None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:49.197376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1ef0d082-6c27-4165-b461-7a4fe936a03d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:49.254927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1105/2175] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:49.322654 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b16862-871c-4eb1-821a-ff054588e12d None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:49.322937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b16862-871c-4eb1-821a-ff054588e12d None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:49.323207 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b16862-871c-4eb1-821a-ff054588e12d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:49.343347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-31b16862-871c-4eb1-821a-ff054588e12d None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:49.344523 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-31b16862-871c-4eb1-821a-ff054588e12d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:49.352569 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1071/2176] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:49.356840 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef0d082-6c27-4165-b461-7a4fe936a03d None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:49.367199 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef0d082-6c27-4165-b461-7a4fe936a03d None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:49.371888 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77a5aad-cf4d-4bc8-aede-c5663f14aa6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:49.372490 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77a5aad-cf4d-4bc8-aede-c5663f14aa6a admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:49.372771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77a5aad-cf4d-4bc8-aede-c5663f14aa6a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:49.373046 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77a5aad-cf4d-4bc8-aede-c5663f14aa6a admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:49.373695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77a5aad-cf4d-4bc8-aede-c5663f14aa6a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:02:49.374000 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77a5aad-cf4d-4bc8-aede-c5663f14aa6a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:49.375617 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77a5aad-cf4d-4bc8-aede-c5663f14aa6a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:49.388478 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1106/2177] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:02:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:02:49.392849 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b16862-871c-4eb1-821a-ff054588e12d None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:49.402848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b16862-871c-4eb1-821a-ff054588e12d None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:49.407487 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08626d5c-00f2-41ec-8964-f43caed3c169 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:49.408113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08626d5c-00f2-41ec-8964-f43caed3c169 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:49.408390 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08626d5c-00f2-41ec-8964-f43caed3c169 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:49.408662 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08626d5c-00f2-41ec-8964-f43caed3c169 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:49.409283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08626d5c-00f2-41ec-8964-f43caed3c169 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:49.411165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08626d5c-00f2-41ec-8964-f43caed3c169 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:49.442799 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1072/2178] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:49 2022] POST /identity/v3/projects => generated 367 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:49.448165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77a5aad-cf4d-4bc8-aede-c5663f14aa6a admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:49.458487 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77a5aad-cf4d-4bc8-aede-c5663f14aa6a admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:49.524369 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e849f67-45b8-43f1-bf9b-bb9b35eaf20a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:49.525057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e849f67-45b8-43f1-bf9b-bb9b35eaf20a admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:49.525346 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e849f67-45b8-43f1-bf9b-bb9b35eaf20a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:49.525620 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e849f67-45b8-43f1-bf9b-bb9b35eaf20a admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:49.526257 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e849f67-45b8-43f1-bf9b-bb9b35eaf20a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:49.527930 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08626d5c-00f2-41ec-8964-f43caed3c169 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:49.529758 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-08626d5c-00f2-41ec-8964-f43caed3c169 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:49.534277 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e849f67-45b8-43f1-bf9b-bb9b35eaf20a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:49.544594 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08626d5c-00f2-41ec-8964-f43caed3c169 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:49.546145 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:49.551760 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bacd66-19dd-4aee-90c0-eab2b6a75630 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:49.552396 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bacd66-19dd-4aee-90c0-eab2b6a75630 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:49.552756 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bacd66-19dd-4aee-90c0-eab2b6a75630 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:49.553106 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bacd66-19dd-4aee-90c0-eab2b6a75630 admin admin] PATH_INFO: `/v3/users/d08c674729f1490597fb54dc197c1d79` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:49.555045 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bacd66-19dd-4aee-90c0-eab2b6a75630 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d08c674729f1490597fb54dc197c1d79)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:49.557199 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bacd66-19dd-4aee-90c0-eab2b6a75630 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:49.559674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1107/2179] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:49 2022] POST /identity/v3/users => generated 284 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:49.563589 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e849f67-45b8-43f1-bf9b-bb9b35eaf20a admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:49.574410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e849f67-45b8-43f1-bf9b-bb9b35eaf20a admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:49.579019 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73efd0d-9864-4185-a908-a658645c7a57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:49.579638 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73efd0d-9864-4185-a908-a658645c7a57 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:49.579921 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73efd0d-9864-4185-a908-a658645c7a57 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:49.580193 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73efd0d-9864-4185-a908-a658645c7a57 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:49.580830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73efd0d-9864-4185-a908-a658645c7a57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:49.582458 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73efd0d-9864-4185-a908-a658645c7a57 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:49.593933 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1108/2180] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:49 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:02:49.597851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73efd0d-9864-4185-a908-a658645c7a57 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:49.605792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-98bacd66-19dd-4aee-90c0-eab2b6a75630 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd08c674729f1490597fb54dc197c1d79'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:49.608791 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73efd0d-9864-4185-a908-a658645c7a57 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:49.612705 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-98bacd66-19dd-4aee-90c0-eab2b6a75630 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd08c674729f1490597fb54dc197c1d79'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:49.613382 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-98bacd66-19dd-4aee-90c0-eab2b6a75630 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd08c674729f1490597fb54dc197c1d79'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:49.613507 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fe99db-a4bc-4e7b-830b-d040458eef17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:49.614006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-98bacd66-19dd-4aee-90c0-eab2b6a75630 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd08c674729f1490597fb54dc197c1d79'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:49.614110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fe99db-a4bc-4e7b-830b-d040458eef17 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:49.614286 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fe99db-a4bc-4e7b-830b-d040458eef17 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:49.614560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fe99db-a4bc-4e7b-830b-d040458eef17 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:49.615247 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fe99db-a4bc-4e7b-830b-d040458eef17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:49.617910 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fe99db-a4bc-4e7b-830b-d040458eef17 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:49.627248 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-98bacd66-19dd-4aee-90c0-eab2b6a75630 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd08c674729f1490597fb54dc197c1d79'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:49.628000 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1109/2181] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:49 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:02:49.632551 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fe99db-a4bc-4e7b-830b-d040458eef17 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:49.634504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-44fe99db-a4bc-4e7b-830b-d040458eef17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:49.641386 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1073/2182] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:49 2022] DELETE /identity/v3/users/d08c674729f1490597fb54dc197c1d79 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:02:49.645722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f38a2f-430b-4810-a3f4-c53b8d00c8c3 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:49.646010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f38a2f-430b-4810-a3f4-c53b8d00c8c3 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:49.646288 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f38a2f-430b-4810-a3f4-c53b8d00c8c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:49.646943 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fe99db-a4bc-4e7b-830b-d040458eef17 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:49.664743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-e0f38a2f-430b-4810-a3f4-c53b8d00c8c3 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:49.665954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e0f38a2f-430b-4810-a3f4-c53b8d00c8c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:49.704553 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcb611b-682d-4f18-bd25-1a6c0fe6c6b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:49.705294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcb611b-682d-4f18-bd25-1a6c0fe6c6b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:49.705578 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcb611b-682d-4f18-bd25-1a6c0fe6c6b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:49.705849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcb611b-682d-4f18-bd25-1a6c0fe6c6b1 admin admin] PATH_INFO: `/v3/projects/6144ecc8e2bb4918af3b43abaf431eb0/users/c8c9e3dd67a4491eb7ab10a5a7e8af09/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:49.720405 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcb611b-682d-4f18-bd25-1a6c0fe6c6b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6144ecc8e2bb4918af3b43abaf431eb0, user_id=c8c9e3dd67a4491eb7ab10a5a7e8af09, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:49.723070 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcb611b-682d-4f18-bd25-1a6c0fe6c6b1 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:49.726154 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1074/2183] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:02:49.736395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f38a2f-430b-4810-a3f4-c53b8d00c8c3 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:49.736659 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1110/2184] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:49 2022] PUT /identity/v3/projects/6144ecc8e2bb4918af3b43abaf431eb0/users/c8c9e3dd67a4491eb7ab10a5a7e8af09/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:02:49.737794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e0f38a2f-430b-4810-a3f4-c53b8d00c8c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:49.748392 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f15fed-bdd4-4516-b489-2aca64ed14ce None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:49.748681 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f15fed-bdd4-4516-b489-2aca64ed14ce None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:49.748959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f15fed-bdd4-4516-b489-2aca64ed14ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:49.758706 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:DELETE=14,keystone:SELECT=83,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:49.777440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-46f15fed-bdd4-4516-b489-2aca64ed14ce None None] MFA Rules not processed for user `c8c9e3dd67a4491eb7ab10a5a7e8af09`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:49.778654 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-46f15fed-bdd4-4516-b489-2aca64ed14ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:49.798977 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f38a2f-430b-4810-a3f4-c53b8d00c8c3 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:49.843867 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1111/2185] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:49 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:02:49.849510 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926554ea-d8ad-4dcf-9e47-678d820208fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:49.850508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926554ea-d8ad-4dcf-9e47-678d820208fa service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:49.850801 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926554ea-d8ad-4dcf-9e47-678d820208fa service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:49.851280 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926554ea-d8ad-4dcf-9e47-678d820208fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:49.856558 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f15fed-bdd4-4516-b489-2aca64ed14ce None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:49.857910 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-46f15fed-bdd4-4516-b489-2aca64ed14ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:49.913303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926554ea-d8ad-4dcf-9e47-678d820208fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:49.914252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:49.914736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926554ea-d8ad-4dcf-9e47-678d820208fa service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:49.920330 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f15fed-bdd4-4516-b489-2aca64ed14ce None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:49.924380 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf9d33f-5acf-4414-9bbb-1a973e75b0c7 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9808dcba0b64027bb70e31d1569957d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:49.925138 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1075/2186] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 15:02:49.925288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf9d33f-5acf-4414-9bbb-1a973e75b0c7 service glance] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:49.925397 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf9d33f-5acf-4414-9bbb-1a973e75b0c7 service glance] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:49.925601 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf9d33f-5acf-4414-9bbb-1a973e75b0c7 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:49.934822 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf9d33f-5acf-4414-9bbb-1a973e75b0c7 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:49.936306 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf9d33f-5acf-4414-9bbb-1a973e75b0c7 service glance] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:49.946181 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1112/2187] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:49 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:50.219657 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926554ea-d8ad-4dcf-9e47-678d820208fa service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:50.221804 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-926554ea-d8ad-4dcf-9e47-678d820208fa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:50.235910 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926554ea-d8ad-4dcf-9e47-678d820208fa service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:50.242063 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080b050a-6a6b-4ee4-a104-493fdd8c26a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:50.242817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080b050a-6a6b-4ee4-a104-493fdd8c26a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:50.243101 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080b050a-6a6b-4ee4-a104-493fdd8c26a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:50.243371 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080b050a-6a6b-4ee4-a104-493fdd8c26a2 admin admin] PATH_INFO: `/v3/projects/7762aed01fe74afdb22fb68970903942` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:50.244580 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080b050a-6a6b-4ee4-a104-493fdd8c26a2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7762aed01fe74afdb22fb68970903942)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:50.246480 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080b050a-6a6b-4ee4-a104-493fdd8c26a2 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:50.316261 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-080b050a-6a6b-4ee4-a104-493fdd8c26a2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7762aed01fe74afdb22fb68970903942', 'request_id': 'req-080b050a-6a6b-4ee4-a104-493fdd8c26a2'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:50.325898 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-080b050a-6a6b-4ee4-a104-493fdd8c26a2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7762aed01fe74afdb22fb68970903942', 'request_id': 'req-080b050a-6a6b-4ee4-a104-493fdd8c26a2'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:50.332670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-080b050a-6a6b-4ee4-a104-493fdd8c26a2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7762aed01fe74afdb22fb68970903942', 'request_id': 'req-080b050a-6a6b-4ee4-a104-493fdd8c26a2'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:50.345021 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-080b050a-6a6b-4ee4-a104-493fdd8c26a2 admin admin] The token cache is being invalidate because project 7762aed01fe74afdb22fb68970903942 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:50.345313 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-080b050a-6a6b-4ee4-a104-493fdd8c26a2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:50.346086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-080b050a-6a6b-4ee4-a104-493fdd8c26a2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:50.347903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1076/2188] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:50 2022] DELETE /identity/v3/projects/7762aed01fe74afdb22fb68970903942 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:50.352234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf9d33f-5acf-4414-9bbb-1a973e75b0c7 service glance] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:50.364194 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf9d33f-5acf-4414-9bbb-1a973e75b0c7 service glance] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:50.433694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f99b34-dc06-4dae-9b68-1223b54255cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:50.434563 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f99b34-dc06-4dae-9b68-1223b54255cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:50.434876 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f99b34-dc06-4dae-9b68-1223b54255cf admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:50.435154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f99b34-dc06-4dae-9b68-1223b54255cf admin admin] PATH_INFO: `/v3/users/45d026015d6149048ffb10b5feef847b` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:50.436348 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f99b34-dc06-4dae-9b68-1223b54255cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=45d026015d6149048ffb10b5feef847b)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:50.439206 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f99b34-dc06-4dae-9b68-1223b54255cf admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:50.499612 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a9f99b34-dc06-4dae-9b68-1223b54255cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '45d026015d6149048ffb10b5feef847b'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:50.500252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a9f99b34-dc06-4dae-9b68-1223b54255cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '45d026015d6149048ffb10b5feef847b'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:50.512999 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a9f99b34-dc06-4dae-9b68-1223b54255cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '45d026015d6149048ffb10b5feef847b'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:50.520312 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a9f99b34-dc06-4dae-9b68-1223b54255cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '45d026015d6149048ffb10b5feef847b'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:50.528100 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a9f99b34-dc06-4dae-9b68-1223b54255cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '45d026015d6149048ffb10b5feef847b'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:50.534635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1113/2189] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:50 2022] DELETE /identity/v3/users/45d026015d6149048ffb10b5feef847b => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:50.827030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080b050a-6a6b-4ee4-a104-493fdd8c26a2 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:50.828581 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-080b050a-6a6b-4ee4-a104-493fdd8c26a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:50.844875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080b050a-6a6b-4ee4-a104-493fdd8c26a2 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:50.849316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:50.916332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40ffd25-bb70-47f1-86f9-ea1ad6442b25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:50.917046 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40ffd25-bb70-47f1-86f9-ea1ad6442b25 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:50.917327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40ffd25-bb70-47f1-86f9-ea1ad6442b25 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:50.917627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40ffd25-bb70-47f1-86f9-ea1ad6442b25 admin admin] PATH_INFO: `/v3/projects/9c7f43c375e34c229ebf8d826d6cd88b` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:50.918767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40ffd25-bb70-47f1-86f9-ea1ad6442b25 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9c7f43c375e34c229ebf8d826d6cd88b)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:50.920709 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40ffd25-bb70-47f1-86f9-ea1ad6442b25 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:50.970443 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c40ffd25-bb70-47f1-86f9-ea1ad6442b25 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9c7f43c375e34c229ebf8d826d6cd88b', 'request_id': 'req-c40ffd25-bb70-47f1-86f9-ea1ad6442b25'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:50.977920 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c40ffd25-bb70-47f1-86f9-ea1ad6442b25 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9c7f43c375e34c229ebf8d826d6cd88b', 'request_id': 'req-c40ffd25-bb70-47f1-86f9-ea1ad6442b25'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:50.983575 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c40ffd25-bb70-47f1-86f9-ea1ad6442b25 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9c7f43c375e34c229ebf8d826d6cd88b', 'request_id': 'req-c40ffd25-bb70-47f1-86f9-ea1ad6442b25'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:50.994750 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c40ffd25-bb70-47f1-86f9-ea1ad6442b25 admin admin] The token cache is being invalidate because project 9c7f43c375e34c229ebf8d826d6cd88b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:50.995038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c40ffd25-bb70-47f1-86f9-ea1ad6442b25 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:50.995725 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-c40ffd25-bb70-47f1-86f9-ea1ad6442b25 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:50.997235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1077/2190] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:50 2022] DELETE /identity/v3/projects/9c7f43c375e34c229ebf8d826d6cd88b => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:51.002320 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2d4796-1448-4efb-a1fc-3de19d879dd6 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:51.002601 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2d4796-1448-4efb-a1fc-3de19d879dd6 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:51.002874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2d4796-1448-4efb-a1fc-3de19d879dd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:51.023784 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-fb2d4796-1448-4efb-a1fc-3de19d879dd6 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:51.024988 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-fb2d4796-1448-4efb-a1fc-3de19d879dd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:51.084151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1114/2191] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:02:51.093866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ca9362-e5a6-4da4-858b-89068d0da707 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:51.094150 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ca9362-e5a6-4da4-858b-89068d0da707 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:51.094424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ca9362-e5a6-4da4-858b-89068d0da707 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:51.114115 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-15ca9362-e5a6-4da4-858b-89068d0da707 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:51.115320 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-15ca9362-e5a6-4da4-858b-89068d0da707 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:51.124033 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1078/2192] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:51.128626 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2d4796-1448-4efb-a1fc-3de19d879dd6 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:51.139202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2d4796-1448-4efb-a1fc-3de19d879dd6 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:51.143759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ec4a6f-d032-4873-8fdc-694f07a62ba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:51.144353 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ec4a6f-d032-4873-8fdc-694f07a62ba1 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:51.144634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ec4a6f-d032-4873-8fdc-694f07a62ba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:51.144906 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ec4a6f-d032-4873-8fdc-694f07a62ba1 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:51.145547 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ec4a6f-d032-4873-8fdc-694f07a62ba1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:02:51.145848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ec4a6f-d032-4873-8fdc-694f07a62ba1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:51.147699 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ec4a6f-d032-4873-8fdc-694f07a62ba1 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:51.150022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:51.161242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1115/2193] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:02:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:02:51.166303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ca9362-e5a6-4da4-858b-89068d0da707 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:51.175763 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ca9362-e5a6-4da4-858b-89068d0da707 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:51.181510 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0946352e-221b-49ff-a8bf-601470d71bbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:51.182106 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0946352e-221b-49ff-a8bf-601470d71bbc admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:51.182388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0946352e-221b-49ff-a8bf-601470d71bbc admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:51.182664 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0946352e-221b-49ff-a8bf-601470d71bbc admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:51.183308 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0946352e-221b-49ff-a8bf-601470d71bbc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:51.185265 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0946352e-221b-49ff-a8bf-601470d71bbc admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:51.214678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1079/2194] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:51 2022] POST /identity/v3/projects => generated 379 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:51.218807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ec4a6f-d032-4873-8fdc-694f07a62ba1 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:51.228940 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ec4a6f-d032-4873-8fdc-694f07a62ba1 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:51.292455 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de489d83-e4cb-4897-9709-7e061d8798d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:51.293499 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de489d83-e4cb-4897-9709-7e061d8798d1 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:51.293781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de489d83-e4cb-4897-9709-7e061d8798d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:51.294055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de489d83-e4cb-4897-9709-7e061d8798d1 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:51.294703 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de489d83-e4cb-4897-9709-7e061d8798d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:51.297102 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de489d83-e4cb-4897-9709-7e061d8798d1 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:51.321148 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1116/2195] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:51 2022] POST /identity/v3/users => generated 290 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:51.325612 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0946352e-221b-49ff-a8bf-601470d71bbc admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:51.336577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0946352e-221b-49ff-a8bf-601470d71bbc admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:51.341311 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074e5fa3-ad15-4604-a6c8-6ddc2bcc37fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:51.341915 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074e5fa3-ad15-4604-a6c8-6ddc2bcc37fa admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:51.342200 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074e5fa3-ad15-4604-a6c8-6ddc2bcc37fa admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:51.342478 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074e5fa3-ad15-4604-a6c8-6ddc2bcc37fa admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:51.343136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074e5fa3-ad15-4604-a6c8-6ddc2bcc37fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:51.344806 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074e5fa3-ad15-4604-a6c8-6ddc2bcc37fa admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:51.355657 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1080/2196] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:02:51.359766 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de489d83-e4cb-4897-9709-7e061d8798d1 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:51.370728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de489d83-e4cb-4897-9709-7e061d8798d1 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:51.376276 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b22a22-74e7-428e-92be-ec97ede68241 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:51.376887 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b22a22-74e7-428e-92be-ec97ede68241 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:51.377209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b22a22-74e7-428e-92be-ec97ede68241 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:51.377466 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b22a22-74e7-428e-92be-ec97ede68241 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:51.378106 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b22a22-74e7-428e-92be-ec97ede68241 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:51.379754 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b22a22-74e7-428e-92be-ec97ede68241 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:51.391992 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1117/2197] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:51 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:51.396176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074e5fa3-ad15-4604-a6c8-6ddc2bcc37fa admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:51.406928 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074e5fa3-ad15-4604-a6c8-6ddc2bcc37fa admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:51.412351 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98573f6d-1d9d-43ad-b700-df09a3692618 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:51.413122 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98573f6d-1d9d-43ad-b700-df09a3692618 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:51.413402 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98573f6d-1d9d-43ad-b700-df09a3692618 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:51.413680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98573f6d-1d9d-43ad-b700-df09a3692618 admin admin] PATH_INFO: `/v3/projects/cb2e3e667f5d4fb580780953b08f8201/users/8cfd9f6b77244253bddda985fb81ab79/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:51.429184 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98573f6d-1d9d-43ad-b700-df09a3692618 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb2e3e667f5d4fb580780953b08f8201, user_id=8cfd9f6b77244253bddda985fb81ab79, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:51.431946 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98573f6d-1d9d-43ad-b700-df09a3692618 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:51.445645 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1081/2198] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:51 2022] PUT /identity/v3/projects/cb2e3e667f5d4fb580780953b08f8201/users/8cfd9f6b77244253bddda985fb81ab79/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:51.457785 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77f49b1-b1e7-407d-a116-e9b096414bbc None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:51.458061 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77f49b1-b1e7-407d-a116-e9b096414bbc None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:51.458336 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77f49b1-b1e7-407d-a116-e9b096414bbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:51.488712 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-a77f49b1-b1e7-407d-a116-e9b096414bbc None None] MFA Rules not processed for user `8cfd9f6b77244253bddda985fb81ab79`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:51.490146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a77f49b1-b1e7-407d-a116-e9b096414bbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:51.558388 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1118/2199] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:51 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:02:51.562837 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98573f6d-1d9d-43ad-b700-df09a3692618 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:51.603312 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:51.633889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98573f6d-1d9d-43ad-b700-df09a3692618 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:51.638603 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1780979a-5614-4b3f-b759-790746c730f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:51.639200 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1780979a-5614-4b3f-b759-790746c730f9 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:51.639485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1780979a-5614-4b3f-b759-790746c730f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:51.639785 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1780979a-5614-4b3f-b759-790746c730f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:51.640423 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1780979a-5614-4b3f-b759-790746c730f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:51.642321 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1780979a-5614-4b3f-b759-790746c730f9 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:51.675380 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1082/2200] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:51 2022] POST /identity/v3/projects => generated 375 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:51.676617 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77f49b1-b1e7-407d-a116-e9b096414bbc None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:51.687974 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77f49b1-b1e7-407d-a116-e9b096414bbc None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:51.748270 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68982c2f-0421-47ad-a15b-3ab0329321e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:51.748876 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68982c2f-0421-47ad-a15b-3ab0329321e3 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:51.749153 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68982c2f-0421-47ad-a15b-3ab0329321e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:51.749424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68982c2f-0421-47ad-a15b-3ab0329321e3 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:51.750062 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68982c2f-0421-47ad-a15b-3ab0329321e3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:51.752020 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68982c2f-0421-47ad-a15b-3ab0329321e3 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:51.764475 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:51.780401 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1119/2201] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:51 2022] POST /identity/v3/users => generated 288 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:02:51.781538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1780979a-5614-4b3f-b759-790746c730f9 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:51.791901 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1780979a-5614-4b3f-b759-790746c730f9 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:51.797341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139777d2-b9c1-40b9-bd18-06a89c32199e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:51.797947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139777d2-b9c1-40b9-bd18-06a89c32199e admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:51.798348 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139777d2-b9c1-40b9-bd18-06a89c32199e admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:51.798728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139777d2-b9c1-40b9-bd18-06a89c32199e admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:51.799376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139777d2-b9c1-40b9-bd18-06a89c32199e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:51.801129 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139777d2-b9c1-40b9-bd18-06a89c32199e admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:51.813168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1083/2202] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:51 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:51.817200 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68982c2f-0421-47ad-a15b-3ab0329321e3 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:51.828555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68982c2f-0421-47ad-a15b-3ab0329321e3 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:51.833583 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95641b1-9543-42b5-a981-594cdd765edb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:51.834221 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95641b1-9543-42b5-a981-594cdd765edb admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:51.834544 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95641b1-9543-42b5-a981-594cdd765edb admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:51.835543 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95641b1-9543-42b5-a981-594cdd765edb admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:51.836344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95641b1-9543-42b5-a981-594cdd765edb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:51.838219 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95641b1-9543-42b5-a981-594cdd765edb admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:51.849264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1120/2203] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:51 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:51.853432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139777d2-b9c1-40b9-bd18-06a89c32199e admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:51.865290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139777d2-b9c1-40b9-bd18-06a89c32199e admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:51.871331 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150ad2fa-5ddd-4948-8060-1a7255381b71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:51.872028 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150ad2fa-5ddd-4948-8060-1a7255381b71 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:51.872306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150ad2fa-5ddd-4948-8060-1a7255381b71 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:51.872578 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150ad2fa-5ddd-4948-8060-1a7255381b71 admin admin] PATH_INFO: `/v3/projects/a3a02e30ddb447c1903c07e7b154ab3e/users/5420e6a7d39c43e4b1d17e76452898ea/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:51.888113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150ad2fa-5ddd-4948-8060-1a7255381b71 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a3a02e30ddb447c1903c07e7b154ab3e, user_id=5420e6a7d39c43e4b1d17e76452898ea, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:51.890752 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150ad2fa-5ddd-4948-8060-1a7255381b71 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:51.906385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1084/2204] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:51 2022] PUT /identity/v3/projects/a3a02e30ddb447c1903c07e7b154ab3e/users/5420e6a7d39c43e4b1d17e76452898ea/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:51.916802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82b2a9f-efd7-460c-882b-affacb5d803d None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:51.917506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82b2a9f-efd7-460c-882b-affacb5d803d None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:51.917901 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82b2a9f-efd7-460c-882b-affacb5d803d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:51.949345 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-d82b2a9f-efd7-460c-882b-affacb5d803d None None] MFA Rules not processed for user `5420e6a7d39c43e4b1d17e76452898ea`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:51.950486 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d82b2a9f-efd7-460c-882b-affacb5d803d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:52.017766 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1121/2205] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:51 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:52.022938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150ad2fa-5ddd-4948-8060-1a7255381b71 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:52.091404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150ad2fa-5ddd-4948-8060-1a7255381b71 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:52.096059 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275a6f92-de4a-4220-84fb-1c78bc362123 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:52.096655 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275a6f92-de4a-4220-84fb-1c78bc362123 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:52.096932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275a6f92-de4a-4220-84fb-1c78bc362123 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:52.097243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275a6f92-de4a-4220-84fb-1c78bc362123 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:52.098326 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275a6f92-de4a-4220-84fb-1c78bc362123 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:52.101982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275a6f92-de4a-4220-84fb-1c78bc362123 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:52.130060 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1085/2206] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:52 2022] POST /identity/v3/projects => generated 379 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:52.134999 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82b2a9f-efd7-460c-882b-affacb5d803d None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:52.146425 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82b2a9f-efd7-460c-882b-affacb5d803d None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:52.208698 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ffbc10-e861-4fce-8474-a3df762774e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:52.209225 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ffbc10-e861-4fce-8474-a3df762774e3 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:52.209506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ffbc10-e861-4fce-8474-a3df762774e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:52.209774 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ffbc10-e861-4fce-8474-a3df762774e3 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:52.210447 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ffbc10-e861-4fce-8474-a3df762774e3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:52.212370 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ffbc10-e861-4fce-8474-a3df762774e3 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:52.235410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1122/2207] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:52 2022] POST /identity/v3/users => generated 296 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:52.239438 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275a6f92-de4a-4220-84fb-1c78bc362123 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:52.250418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275a6f92-de4a-4220-84fb-1c78bc362123 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:52.255438 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05a7f66-7709-4951-bc5d-482024f62aaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:52.256177 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05a7f66-7709-4951-bc5d-482024f62aaf admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:52.256374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05a7f66-7709-4951-bc5d-482024f62aaf admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:52.256735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05a7f66-7709-4951-bc5d-482024f62aaf admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:52.257549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05a7f66-7709-4951-bc5d-482024f62aaf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:52.259132 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05a7f66-7709-4951-bc5d-482024f62aaf admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:52.270007 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1086/2208] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:52 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:52.274395 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ffbc10-e861-4fce-8474-a3df762774e3 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:52.284286 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ffbc10-e861-4fce-8474-a3df762774e3 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:52.290382 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564580ad-b011-4c39-845f-1c4966d226ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:52.291211 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564580ad-b011-4c39-845f-1c4966d226ad admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:52.291493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564580ad-b011-4c39-845f-1c4966d226ad admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:52.292054 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564580ad-b011-4c39-845f-1c4966d226ad admin admin] PATH_INFO: `/v3/projects/29d7c9abc45d458ebb165862b8b3eb0e/users/45ffaf67183c4782a2e4282f18891883/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:52.309697 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564580ad-b011-4c39-845f-1c4966d226ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=29d7c9abc45d458ebb165862b8b3eb0e, user_id=45ffaf67183c4782a2e4282f18891883, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:52.314136 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564580ad-b011-4c39-845f-1c4966d226ad admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:52.328557 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1123/2209] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:52 2022] PUT /identity/v3/projects/29d7c9abc45d458ebb165862b8b3eb0e/users/45ffaf67183c4782a2e4282f18891883/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:52.332614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05a7f66-7709-4951-bc5d-482024f62aaf admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:52.342952 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05a7f66-7709-4951-bc5d-482024f62aaf admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:52.396822 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:52.404987 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16afc1a4-53b0-4ae7-acb7-272a94cc305a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:52.405642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16afc1a4-53b0-4ae7-acb7-272a94cc305a admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:52.405918 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16afc1a4-53b0-4ae7-acb7-272a94cc305a admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:52.406187 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16afc1a4-53b0-4ae7-acb7-272a94cc305a admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:52.406834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16afc1a4-53b0-4ae7-acb7-272a94cc305a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:52.408513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16afc1a4-53b0-4ae7-acb7-272a94cc305a admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:52.419668 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1087/2210] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:52 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:52.424003 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564580ad-b011-4c39-845f-1c4966d226ad admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:52.434950 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564580ad-b011-4c39-845f-1c4966d226ad admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:52.439541 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b550d4f-ff45-453d-92fa-3d18369e015c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:52.440221 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b550d4f-ff45-453d-92fa-3d18369e015c admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:52.440500 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b550d4f-ff45-453d-92fa-3d18369e015c admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:52.440776 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b550d4f-ff45-453d-92fa-3d18369e015c admin admin] PATH_INFO: `/v3/projects/29d7c9abc45d458ebb165862b8b3eb0e/users/45ffaf67183c4782a2e4282f18891883/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:52.442699 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b550d4f-ff45-453d-92fa-3d18369e015c admin admin] RBAC: Authorizing `identity:create_grant(project_id=29d7c9abc45d458ebb165862b8b3eb0e, user_id=45ffaf67183c4782a2e4282f18891883, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:52.445348 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b550d4f-ff45-453d-92fa-3d18369e015c admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:52.457951 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1124/2211] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:52 2022] PUT /identity/v3/projects/29d7c9abc45d458ebb165862b8b3eb0e/users/45ffaf67183c4782a2e4282f18891883/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:52.469161 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf66ec4-955c-4f2a-8d7f-2a08f5e6d15b None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:52.469406 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf66ec4-955c-4f2a-8d7f-2a08f5e6d15b None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:52.469674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf66ec4-955c-4f2a-8d7f-2a08f5e6d15b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:52.500154 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-adf66ec4-955c-4f2a-8d7f-2a08f5e6d15b None None] MFA Rules not processed for user `45ffaf67183c4782a2e4282f18891883`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:52.501369 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-adf66ec4-955c-4f2a-8d7f-2a08f5e6d15b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:52.580760 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1088/2212] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:52 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:52.592874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b550d4f-ff45-453d-92fa-3d18369e015c admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:52.594599 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-9b550d4f-ff45-453d-92fa-3d18369e015c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:52.623361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:52.654652 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b550d4f-ff45-453d-92fa-3d18369e015c admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:52.658347 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f1063e-e94a-4be7-82e6-dc592ed6e29e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:52.658947 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f1063e-e94a-4be7-82e6-dc592ed6e29e service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:52.659225 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f1063e-e94a-4be7-82e6-dc592ed6e29e service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:52.659497 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f1063e-e94a-4be7-82e6-dc592ed6e29e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:52.669014 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f1063e-e94a-4be7-82e6-dc592ed6e29e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:52.670685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f1063e-e94a-4be7-82e6-dc592ed6e29e service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:52.681653 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1125/2213] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:52 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:52.727912 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf66ec4-955c-4f2a-8d7f-2a08f5e6d15b None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:52.729326 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-adf66ec4-955c-4f2a-8d7f-2a08f5e6d15b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:52.741958 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf66ec4-955c-4f2a-8d7f-2a08f5e6d15b None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:52.745586 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3973ca-0bcc-457d-8171-9ff7617c6ac1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:52.746175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3973ca-0bcc-457d-8171-9ff7617c6ac1 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:52.746452 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3973ca-0bcc-457d-8171-9ff7617c6ac1 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:52.746722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3973ca-0bcc-457d-8171-9ff7617c6ac1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:52.810203 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3973ca-0bcc-457d-8171-9ff7617c6ac1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:52.811857 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3973ca-0bcc-457d-8171-9ff7617c6ac1 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:52.822179 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1089/2214] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:52 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:55.754765 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f1063e-e94a-4be7-82e6-dc592ed6e29e service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:55.823763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f1063e-e94a-4be7-82e6-dc592ed6e29e service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:55.828147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8603bfa3-30bb-45b6-a109-d7cfffff151c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:55.828768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8603bfa3-30bb-45b6-a109-d7cfffff151c admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:55.829046 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8603bfa3-30bb-45b6-a109-d7cfffff151c admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:55.829317 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8603bfa3-30bb-45b6-a109-d7cfffff151c admin admin] PATH_INFO: `/v3/users/8cfd9f6b77244253bddda985fb81ab79` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:55.830414 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8603bfa3-30bb-45b6-a109-d7cfffff151c admin admin] RBAC: Authorizing `identity:delete_user(user_id=8cfd9f6b77244253bddda985fb81ab79)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:55.832347 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8603bfa3-30bb-45b6-a109-d7cfffff151c admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:55.882609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8603bfa3-30bb-45b6-a109-d7cfffff151c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8cfd9f6b77244253bddda985fb81ab79'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:55.883264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8603bfa3-30bb-45b6-a109-d7cfffff151c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8cfd9f6b77244253bddda985fb81ab79'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:55.895227 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8603bfa3-30bb-45b6-a109-d7cfffff151c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8cfd9f6b77244253bddda985fb81ab79'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:55.903836 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8603bfa3-30bb-45b6-a109-d7cfffff151c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8cfd9f6b77244253bddda985fb81ab79'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:55.911802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8603bfa3-30bb-45b6-a109-d7cfffff151c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8cfd9f6b77244253bddda985fb81ab79'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:55.919208 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1126/2215] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:55 2022] DELETE /identity/v3/users/8cfd9f6b77244253bddda985fb81ab79 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:55.923784 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54fa162-051d-4bb5-89e5-052ed4bdb9dc None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:55.924244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54fa162-051d-4bb5-89e5-052ed4bdb9dc None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:55.924339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54fa162-051d-4bb5-89e5-052ed4bdb9dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:55.945134 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-a54fa162-051d-4bb5-89e5-052ed4bdb9dc None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:55.946618 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a54fa162-051d-4bb5-89e5-052ed4bdb9dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:56.007280 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1090/2216] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:56.017734 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8603bfa3-30bb-45b6-a109-d7cfffff151c admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:56.019421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8603bfa3-30bb-45b6-a109-d7cfffff151c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:56.074440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:56.080503 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8603bfa3-30bb-45b6-a109-d7cfffff151c admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:56.085669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b1bc1f-aab0-4777-abbd-24a7ca9e2dbf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:56.086279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b1bc1f-aab0-4777-abbd-24a7ca9e2dbf service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:56.086560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b1bc1f-aab0-4777-abbd-24a7ca9e2dbf service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:56.086836 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b1bc1f-aab0-4777-abbd-24a7ca9e2dbf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:56.095911 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b1bc1f-aab0-4777-abbd-24a7ca9e2dbf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:56.097757 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b1bc1f-aab0-4777-abbd-24a7ca9e2dbf service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:56.108367 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1127/2217] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:56.386574 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54fa162-051d-4bb5-89e5-052ed4bdb9dc None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:56.388294 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a54fa162-051d-4bb5-89e5-052ed4bdb9dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:56.400423 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54fa162-051d-4bb5-89e5-052ed4bdb9dc None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:56.405023 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80450390-c412-4a7d-91a6-8c25f0240e02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:56.405696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80450390-c412-4a7d-91a6-8c25f0240e02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:56.406156 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80450390-c412-4a7d-91a6-8c25f0240e02 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:56.406636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80450390-c412-4a7d-91a6-8c25f0240e02 admin admin] PATH_INFO: `/v3/projects/cb2e3e667f5d4fb580780953b08f8201` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:56.408699 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80450390-c412-4a7d-91a6-8c25f0240e02 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb2e3e667f5d4fb580780953b08f8201)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:56.411735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80450390-c412-4a7d-91a6-8c25f0240e02 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:56.424768 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:56.463983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-80450390-c412-4a7d-91a6-8c25f0240e02 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb2e3e667f5d4fb580780953b08f8201', 'request_id': 'req-80450390-c412-4a7d-91a6-8c25f0240e02'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:56.471441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-80450390-c412-4a7d-91a6-8c25f0240e02 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb2e3e667f5d4fb580780953b08f8201', 'request_id': 'req-80450390-c412-4a7d-91a6-8c25f0240e02'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:56.477057 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-80450390-c412-4a7d-91a6-8c25f0240e02 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb2e3e667f5d4fb580780953b08f8201', 'request_id': 'req-80450390-c412-4a7d-91a6-8c25f0240e02'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:56.489722 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-80450390-c412-4a7d-91a6-8c25f0240e02 admin admin] The token cache is being invalidate because project cb2e3e667f5d4fb580780953b08f8201 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:56.490007 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-80450390-c412-4a7d-91a6-8c25f0240e02 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:56.490637 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-80450390-c412-4a7d-91a6-8c25f0240e02 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:56.492841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1091/2218] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:56 2022] DELETE /identity/v3/projects/cb2e3e667f5d4fb580780953b08f8201 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:56.497412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b1bc1f-aab0-4777-abbd-24a7ca9e2dbf service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:56.508193 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b1bc1f-aab0-4777-abbd-24a7ca9e2dbf service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:56.567232 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1fd51f-6631-4e95-bf0a-d56ceb540008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:56.567910 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1fd51f-6631-4e95-bf0a-d56ceb540008 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:56.568193 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1fd51f-6631-4e95-bf0a-d56ceb540008 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:56.568468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1fd51f-6631-4e95-bf0a-d56ceb540008 admin admin] PATH_INFO: `/v3/users/5420e6a7d39c43e4b1d17e76452898ea` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:56.570526 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1fd51f-6631-4e95-bf0a-d56ceb540008 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5420e6a7d39c43e4b1d17e76452898ea)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:56.572733 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1fd51f-6631-4e95-bf0a-d56ceb540008 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:56.624975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-be1fd51f-6631-4e95-bf0a-d56ceb540008 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5420e6a7d39c43e4b1d17e76452898ea'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:56.625737 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-be1fd51f-6631-4e95-bf0a-d56ceb540008 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5420e6a7d39c43e4b1d17e76452898ea'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:56.647101 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-be1fd51f-6631-4e95-bf0a-d56ceb540008 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5420e6a7d39c43e4b1d17e76452898ea'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:56.654303 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-be1fd51f-6631-4e95-bf0a-d56ceb540008 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5420e6a7d39c43e4b1d17e76452898ea'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:56.662556 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-be1fd51f-6631-4e95-bf0a-d56ceb540008 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5420e6a7d39c43e4b1d17e76452898ea'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:56.669734 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1128/2219] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:56 2022] DELETE /identity/v3/users/5420e6a7d39c43e4b1d17e76452898ea => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:57.000256 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80450390-c412-4a7d-91a6-8c25f0240e02 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:57.001784 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-80450390-c412-4a7d-91a6-8c25f0240e02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:57.016235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80450390-c412-4a7d-91a6-8c25f0240e02 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:57.075320 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:57.075983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:57.076269 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:57.076545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16 admin admin] PATH_INFO: `/v3/projects/a3a02e30ddb447c1903c07e7b154ab3e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:57.077668 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a3a02e30ddb447c1903c07e7b154ab3e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:57.079665 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:57.127721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a3a02e30ddb447c1903c07e7b154ab3e', 'request_id': 'req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:57.135625 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a3a02e30ddb447c1903c07e7b154ab3e', 'request_id': 'req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:57.141075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a3a02e30ddb447c1903c07e7b154ab3e', 'request_id': 'req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:57.152242 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16 admin admin] The token cache is being invalidate because project a3a02e30ddb447c1903c07e7b154ab3e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:57.152530 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:57.153147 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:57.154617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1092/2220] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:56 2022] DELETE /identity/v3/projects/a3a02e30ddb447c1903c07e7b154ab3e => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:57.158641 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1fd51f-6631-4e95-bf0a-d56ceb540008 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:57.160129 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-be1fd51f-6631-4e95-bf0a-d56ceb540008 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:57.175048 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1fd51f-6631-4e95-bf0a-d56ceb540008 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:57.236291 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493fc541-f513-450f-8774-417da94f3884 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:57.236911 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493fc541-f513-450f-8774-417da94f3884 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:57.237188 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493fc541-f513-450f-8774-417da94f3884 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:57.237457 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493fc541-f513-450f-8774-417da94f3884 admin admin] PATH_INFO: `/v3/users/45ffaf67183c4782a2e4282f18891883` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:57.238550 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493fc541-f513-450f-8774-417da94f3884 admin admin] RBAC: Authorizing `identity:delete_user(user_id=45ffaf67183c4782a2e4282f18891883)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:57.240479 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493fc541-f513-450f-8774-417da94f3884 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:57.251203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:57.289643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-493fc541-f513-450f-8774-417da94f3884 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '45ffaf67183c4782a2e4282f18891883'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:57.290888 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-493fc541-f513-450f-8774-417da94f3884 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '45ffaf67183c4782a2e4282f18891883'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:57.303141 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-493fc541-f513-450f-8774-417da94f3884 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '45ffaf67183c4782a2e4282f18891883'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:57.312900 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-493fc541-f513-450f-8774-417da94f3884 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '45ffaf67183c4782a2e4282f18891883'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:57.321500 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-493fc541-f513-450f-8774-417da94f3884 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '45ffaf67183c4782a2e4282f18891883'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:57.327758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1129/2221] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:57 2022] DELETE /identity/v3/users/45ffaf67183c4782a2e4282f18891883 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:57.603531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:57.605079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:57.618433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90dfcb0-1acc-4e1e-a7be-c9fed035cd16 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:57.649636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:57.680656 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dece202-5962-4cb0-9f6e-e5ea0e34f987 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:57.681211 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dece202-5962-4cb0-9f6e-e5ea0e34f987 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:57.681496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dece202-5962-4cb0-9f6e-e5ea0e34f987 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:57.681775 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dece202-5962-4cb0-9f6e-e5ea0e34f987 admin admin] PATH_INFO: `/v3/projects/29d7c9abc45d458ebb165862b8b3eb0e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:57.682917 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dece202-5962-4cb0-9f6e-e5ea0e34f987 admin admin] RBAC: Authorizing `identity:delete_project(project_id=29d7c9abc45d458ebb165862b8b3eb0e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:57.684956 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dece202-5962-4cb0-9f6e-e5ea0e34f987 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:57.732338 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0dece202-5962-4cb0-9f6e-e5ea0e34f987 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '29d7c9abc45d458ebb165862b8b3eb0e', 'request_id': 'req-0dece202-5962-4cb0-9f6e-e5ea0e34f987'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:57.739954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0dece202-5962-4cb0-9f6e-e5ea0e34f987 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '29d7c9abc45d458ebb165862b8b3eb0e', 'request_id': 'req-0dece202-5962-4cb0-9f6e-e5ea0e34f987'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:57.746142 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0dece202-5962-4cb0-9f6e-e5ea0e34f987 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '29d7c9abc45d458ebb165862b8b3eb0e', 'request_id': 'req-0dece202-5962-4cb0-9f6e-e5ea0e34f987'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:57.757656 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0dece202-5962-4cb0-9f6e-e5ea0e34f987 admin admin] The token cache is being invalidate because project 29d7c9abc45d458ebb165862b8b3eb0e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:02:57.757943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0dece202-5962-4cb0-9f6e-e5ea0e34f987 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:57.758581 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0dece202-5962-4cb0-9f6e-e5ea0e34f987 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:57.760123 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1093/2222] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:02:57 2022] DELETE /identity/v3/projects/29d7c9abc45d458ebb165862b8b3eb0e => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:57.766356 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be66082c-dc92-4977-ad0b-32beae5a8dc0 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:57.766642 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be66082c-dc92-4977-ad0b-32beae5a8dc0 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:57.766915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be66082c-dc92-4977-ad0b-32beae5a8dc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:57.787559 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-be66082c-dc92-4977-ad0b-32beae5a8dc0 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:57.789725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-be66082c-dc92-4977-ad0b-32beae5a8dc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:57.854809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1130/2223] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:57.863236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f21cbcf-b211-4218-81fb-870ad5eb1597 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:57.863525 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f21cbcf-b211-4218-81fb-870ad5eb1597 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:57.863833 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f21cbcf-b211-4218-81fb-870ad5eb1597 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:57.884372 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-6f21cbcf-b211-4218-81fb-870ad5eb1597 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:57.885829 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6f21cbcf-b211-4218-81fb-870ad5eb1597 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:57.894091 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1094/2224] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:57.899062 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be66082c-dc92-4977-ad0b-32beae5a8dc0 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:57.908830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be66082c-dc92-4977-ad0b-32beae5a8dc0 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:57.914062 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15a482c-2355-46b4-833a-e3a5de60a4a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:57.914654 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15a482c-2355-46b4-833a-e3a5de60a4a8 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:57.914936 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15a482c-2355-46b4-833a-e3a5de60a4a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:57.915209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15a482c-2355-46b4-833a-e3a5de60a4a8 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:57.915885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15a482c-2355-46b4-833a-e3a5de60a4a8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:02:57.916192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15a482c-2355-46b4-833a-e3a5de60a4a8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:57.917788 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15a482c-2355-46b4-833a-e3a5de60a4a8 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:57.929501 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1131/2225] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:02:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 15:02:57.934159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f21cbcf-b211-4218-81fb-870ad5eb1597 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:57.945662 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f21cbcf-b211-4218-81fb-870ad5eb1597 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:57.950661 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e9e639-3391-43cb-9b43-202859eb7de0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:57.951272 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e9e639-3391-43cb-9b43-202859eb7de0 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:57.951592 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e9e639-3391-43cb-9b43-202859eb7de0 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:57.951865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e9e639-3391-43cb-9b43-202859eb7de0 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:57.952538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e9e639-3391-43cb-9b43-202859eb7de0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:57.954452 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e9e639-3391-43cb-9b43-202859eb7de0 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:57.987972 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1095/2226] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:57 2022] POST /identity/v3/projects => generated 383 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:57.992323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15a482c-2355-46b4-833a-e3a5de60a4a8 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:58.002895 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15a482c-2355-46b4-833a-e3a5de60a4a8 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:58.065963 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afb63c8-2177-4823-a9fe-652d851199f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:58.066574 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afb63c8-2177-4823-a9fe-652d851199f9 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:58.066853 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afb63c8-2177-4823-a9fe-652d851199f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:58.067125 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afb63c8-2177-4823-a9fe-652d851199f9 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:58.067797 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afb63c8-2177-4823-a9fe-652d851199f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:58.069921 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afb63c8-2177-4823-a9fe-652d851199f9 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:58.093913 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1132/2227] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:57 2022] POST /identity/v3/users => generated 292 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:58.098059 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e9e639-3391-43cb-9b43-202859eb7de0 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:58.109055 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e9e639-3391-43cb-9b43-202859eb7de0 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:58.113843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1373cc-51c8-463f-b5af-32ec9f3d452b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:58.114445 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1373cc-51c8-463f-b5af-32ec9f3d452b admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:58.114727 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1373cc-51c8-463f-b5af-32ec9f3d452b admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:58.115003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1373cc-51c8-463f-b5af-32ec9f3d452b admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:58.115752 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1373cc-51c8-463f-b5af-32ec9f3d452b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:58.117403 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1373cc-51c8-463f-b5af-32ec9f3d452b admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:58.132643 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1096/2228] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:58.133826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afb63c8-2177-4823-a9fe-652d851199f9 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:58.140643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:58.145196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afb63c8-2177-4823-a9fe-652d851199f9 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:58.152309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b26b58-d013-4cf6-8469-d9a63e314326 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:58.153242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b26b58-d013-4cf6-8469-d9a63e314326 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:58.153779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b26b58-d013-4cf6-8469-d9a63e314326 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:58.154704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b26b58-d013-4cf6-8469-d9a63e314326 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:58.155644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b26b58-d013-4cf6-8469-d9a63e314326 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:58.157635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b26b58-d013-4cf6-8469-d9a63e314326 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:58.172152 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1133/2229] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:58 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:58.176305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1373cc-51c8-463f-b5af-32ec9f3d452b admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:58.187590 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1373cc-51c8-463f-b5af-32ec9f3d452b admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:58.192355 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5489ee30-a82a-43f7-b40b-b8aa830e3bda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:58.193018 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5489ee30-a82a-43f7-b40b-b8aa830e3bda admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:58.193290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5489ee30-a82a-43f7-b40b-b8aa830e3bda admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:58.193562 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5489ee30-a82a-43f7-b40b-b8aa830e3bda admin admin] PATH_INFO: `/v3/projects/0891a510c6cb4f438e4ec0a5a98872de/users/9c997ad98afc4893b8030c3a95bb2cf2/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:58.210162 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5489ee30-a82a-43f7-b40b-b8aa830e3bda admin admin] RBAC: Authorizing `identity:create_grant(project_id=0891a510c6cb4f438e4ec0a5a98872de, user_id=9c997ad98afc4893b8030c3a95bb2cf2, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:58.213019 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5489ee30-a82a-43f7-b40b-b8aa830e3bda admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:58.227097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1097/2230] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:58 2022] PUT /identity/v3/projects/0891a510c6cb4f438e4ec0a5a98872de/users/9c997ad98afc4893b8030c3a95bb2cf2/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:58.239493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2509ddc2-1ad2-4cbd-bc7e-c18a765876cf None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:58.240419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2509ddc2-1ad2-4cbd-bc7e-c18a765876cf None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:58.240565 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2509ddc2-1ad2-4cbd-bc7e-c18a765876cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:58.274186 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-2509ddc2-1ad2-4cbd-bc7e-c18a765876cf None None] MFA Rules not processed for user `9c997ad98afc4893b8030c3a95bb2cf2`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:58.276035 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2509ddc2-1ad2-4cbd-bc7e-c18a765876cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:58.362874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1134/2231] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:58 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:58.367579 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5489ee30-a82a-43f7-b40b-b8aa830e3bda admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:58.430740 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:58.434835 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5489ee30-a82a-43f7-b40b-b8aa830e3bda admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:58.441046 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e904bc2-18af-4364-b1ee-82694567f1e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:58.441662 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e904bc2-18af-4364-b1ee-82694567f1e9 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:58.443684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e904bc2-18af-4364-b1ee-82694567f1e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:58.443684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e904bc2-18af-4364-b1ee-82694567f1e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:58.443684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e904bc2-18af-4364-b1ee-82694567f1e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:58.446498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e904bc2-18af-4364-b1ee-82694567f1e9 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:58.473827 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1098/2232] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:02:58 2022] POST /identity/v3/projects => generated 383 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:02:58.478567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2509ddc2-1ad2-4cbd-bc7e-c18a765876cf None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:58.489494 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2509ddc2-1ad2-4cbd-bc7e-c18a765876cf None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:58.548634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0bfb5b-8351-4fe2-96e1-f1d308078f8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:58.549246 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0bfb5b-8351-4fe2-96e1-f1d308078f8e admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:58.549524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0bfb5b-8351-4fe2-96e1-f1d308078f8e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:58.549796 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0bfb5b-8351-4fe2-96e1-f1d308078f8e admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:58.550452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d0bfb5b-8351-4fe2-96e1-f1d308078f8e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:58.552393 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d0bfb5b-8351-4fe2-96e1-f1d308078f8e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:58.576808 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1135/2233] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:02:58 2022] POST /identity/v3/users => generated 298 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:02:58.581475 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e904bc2-18af-4364-b1ee-82694567f1e9 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:58.593038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e904bc2-18af-4364-b1ee-82694567f1e9 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:58.597700 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea297f90-0cbc-4702-a1fa-1b1dc4b458c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:58.598297 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea297f90-0cbc-4702-a1fa-1b1dc4b458c4 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:58.598582 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea297f90-0cbc-4702-a1fa-1b1dc4b458c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:58.598859 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea297f90-0cbc-4702-a1fa-1b1dc4b458c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:58.599533 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea297f90-0cbc-4702-a1fa-1b1dc4b458c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:58.602197 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea297f90-0cbc-4702-a1fa-1b1dc4b458c4 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:58.612974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1099/2234] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:02:58.617078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0bfb5b-8351-4fe2-96e1-f1d308078f8e admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:58.626860 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0bfb5b-8351-4fe2-96e1-f1d308078f8e admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:58.631652 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8582109d-9314-4f44-95e4-30d28c6340ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:58.632391 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8582109d-9314-4f44-95e4-30d28c6340ae admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:58.632671 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8582109d-9314-4f44-95e4-30d28c6340ae admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:58.632945 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8582109d-9314-4f44-95e4-30d28c6340ae admin admin] PATH_INFO: `/v3/projects/9f1e83d9daa7453fb8610b3f058a2b3e/users/7c30ae6f2a5d4a17a92fba80e598569c/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:58.648980 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8582109d-9314-4f44-95e4-30d28c6340ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f1e83d9daa7453fb8610b3f058a2b3e, user_id=7c30ae6f2a5d4a17a92fba80e598569c, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:58.651837 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8582109d-9314-4f44-95e4-30d28c6340ae admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:58.666655 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1136/2235] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:58 2022] PUT /identity/v3/projects/9f1e83d9daa7453fb8610b3f058a2b3e/users/7c30ae6f2a5d4a17a92fba80e598569c/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:58.670691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea297f90-0cbc-4702-a1fa-1b1dc4b458c4 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:58.681688 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea297f90-0cbc-4702-a1fa-1b1dc4b458c4 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:58.743407 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edffc4e-77ef-40cd-b6b1-bf8b747f09a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:58.743974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edffc4e-77ef-40cd-b6b1-bf8b747f09a9 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:58.744258 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edffc4e-77ef-40cd-b6b1-bf8b747f09a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:58.744537 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edffc4e-77ef-40cd-b6b1-bf8b747f09a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:58.745197 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edffc4e-77ef-40cd-b6b1-bf8b747f09a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:58.746806 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edffc4e-77ef-40cd-b6b1-bf8b747f09a9 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:58.758782 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1100/2236] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:02:58 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:02:58.763105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8582109d-9314-4f44-95e4-30d28c6340ae admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:58.773495 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8582109d-9314-4f44-95e4-30d28c6340ae admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:58.778034 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b570396b-83cd-463f-8d24-b704b21d3886 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:58.778707 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b570396b-83cd-463f-8d24-b704b21d3886 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:58.778984 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b570396b-83cd-463f-8d24-b704b21d3886 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:58.779259 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b570396b-83cd-463f-8d24-b704b21d3886 admin admin] PATH_INFO: `/v3/projects/9f1e83d9daa7453fb8610b3f058a2b3e/users/7c30ae6f2a5d4a17a92fba80e598569c/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:58.781228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b570396b-83cd-463f-8d24-b704b21d3886 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f1e83d9daa7453fb8610b3f058a2b3e, user_id=7c30ae6f2a5d4a17a92fba80e598569c, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:58.784301 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b570396b-83cd-463f-8d24-b704b21d3886 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:58.794857 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1137/2237] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:02:58 2022] PUT /identity/v3/projects/9f1e83d9daa7453fb8610b3f058a2b3e/users/7c30ae6f2a5d4a17a92fba80e598569c/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:58.805686 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6966a1f-a8b6-4eda-87d0-afa8babfd6f7 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:58.805976 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6966a1f-a8b6-4eda-87d0-afa8babfd6f7 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:58.806280 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6966a1f-a8b6-4eda-87d0-afa8babfd6f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:58.835267 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-e6966a1f-a8b6-4eda-87d0-afa8babfd6f7 None None] MFA Rules not processed for user `7c30ae6f2a5d4a17a92fba80e598569c`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:58.836512 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e6966a1f-a8b6-4eda-87d0-afa8babfd6f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:58.910468 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1101/2238] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:58 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:58.923772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b570396b-83cd-463f-8d24-b704b21d3886 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:58.923772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b570396b-83cd-463f-8d24-b704b21d3886 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:58.965362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:58.984639 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b570396b-83cd-463f-8d24-b704b21d3886 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:58.988352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021a5953-c774-476d-ac8b-d2612632d226 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:58.988953 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021a5953-c774-476d-ac8b-d2612632d226 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:58.989231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021a5953-c774-476d-ac8b-d2612632d226 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:58.989500 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021a5953-c774-476d-ac8b-d2612632d226 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:58.999098 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021a5953-c774-476d-ac8b-d2612632d226 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:59.000466 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021a5953-c774-476d-ac8b-d2612632d226 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:59.010783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1138/2239] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:58 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:59.102898 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6966a1f-a8b6-4eda-87d0-afa8babfd6f7 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:59.112495 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6966a1f-a8b6-4eda-87d0-afa8babfd6f7 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:59.118083 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa26512-1105-4647-8251-89eb6ebaaded service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:59.118705 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa26512-1105-4647-8251-89eb6ebaaded service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:59.118986 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa26512-1105-4647-8251-89eb6ebaaded service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:59.119262 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa26512-1105-4647-8251-89eb6ebaaded service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:59.169857 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:59.179976 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa26512-1105-4647-8251-89eb6ebaaded service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:59.181419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa26512-1105-4647-8251-89eb6ebaaded service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:59.190451 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1102/2240] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:59 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:02:59.471051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021a5953-c774-476d-ac8b-d2612632d226 service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:59.534283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021a5953-c774-476d-ac8b-d2612632d226 service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:59.539122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2846159-fee5-4e6a-9305-26918f1f6957 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:59.539828 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2846159-fee5-4e6a-9305-26918f1f6957 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:59.540111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2846159-fee5-4e6a-9305-26918f1f6957 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:59.540383 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2846159-fee5-4e6a-9305-26918f1f6957 admin admin] PATH_INFO: `/v3/users/9c997ad98afc4893b8030c3a95bb2cf2` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:59.541501 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2846159-fee5-4e6a-9305-26918f1f6957 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9c997ad98afc4893b8030c3a95bb2cf2)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:59.543447 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2846159-fee5-4e6a-9305-26918f1f6957 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:59.592576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f2846159-fee5-4e6a-9305-26918f1f6957 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9c997ad98afc4893b8030c3a95bb2cf2'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:59.593249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f2846159-fee5-4e6a-9305-26918f1f6957 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9c997ad98afc4893b8030c3a95bb2cf2'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:59.605674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f2846159-fee5-4e6a-9305-26918f1f6957 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9c997ad98afc4893b8030c3a95bb2cf2'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:59.613105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f2846159-fee5-4e6a-9305-26918f1f6957 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9c997ad98afc4893b8030c3a95bb2cf2'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:59.621079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f2846159-fee5-4e6a-9305-26918f1f6957 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9c997ad98afc4893b8030c3a95bb2cf2'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:02:59.627354 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1139/2241] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:02:59 2022] DELETE /identity/v3/users/9c997ad98afc4893b8030c3a95bb2cf2 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:02:59.632058 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ca809a-3e13-4530-992f-1b5e72131e97 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:59.632345 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ca809a-3e13-4530-992f-1b5e72131e97 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:59.632620 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ca809a-3e13-4530-992f-1b5e72131e97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:59.653951 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-e5ca809a-3e13-4530-992f-1b5e72131e97 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:02:59.655175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e5ca809a-3e13-4530-992f-1b5e72131e97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:59.715043 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1103/2242] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:02:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:02:59.724716 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2846159-fee5-4e6a-9305-26918f1f6957 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:02:59.727102 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-f2846159-fee5-4e6a-9305-26918f1f6957 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:02:59.785582 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2846159-fee5-4e6a-9305-26918f1f6957 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:02:59.789859 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4285bb45-b398-4098-a0a3-ce2389512e3e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:02:59.790468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4285bb45-b398-4098-a0a3-ce2389512e3e service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:02:59.790753 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4285bb45-b398-4098-a0a3-ce2389512e3e service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:02:59.791030 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4285bb45-b398-4098-a0a3-ce2389512e3e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:02:59.800398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4285bb45-b398-4098-a0a3-ce2389512e3e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:02:59.801865 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4285bb45-b398-4098-a0a3-ce2389512e3e service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:02:59.808002 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:02:59.813362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1140/2243] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:02:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:03:00.098750 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ca809a-3e13-4530-992f-1b5e72131e97 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:00.100817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e5ca809a-3e13-4530-992f-1b5e72131e97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:00.118316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ca809a-3e13-4530-992f-1b5e72131e97 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:00.124075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:00.124738 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:00.124990 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:00.125367 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb admin admin] PATH_INFO: `/v3/projects/0891a510c6cb4f438e4ec0a5a98872de` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:00.126369 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb admin admin] RBAC: Authorizing `identity:delete_project(project_id=0891a510c6cb4f438e4ec0a5a98872de)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:00.128354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:00.176420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0891a510c6cb4f438e4ec0a5a98872de', 'request_id': 'req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:00.184307 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0891a510c6cb4f438e4ec0a5a98872de', 'request_id': 'req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:00.190159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0891a510c6cb4f438e4ec0a5a98872de', 'request_id': 'req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:00.202877 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb admin admin] The token cache is being invalidate because project 0891a510c6cb4f438e4ec0a5a98872de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:03:00.203165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:00.203881 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:00.205365 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1104/2244] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:03:00 2022] DELETE /identity/v3/projects/0891a510c6cb4f438e4ec0a5a98872de => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:03:00.210069 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4285bb45-b398-4098-a0a3-ce2389512e3e service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:00.220093 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4285bb45-b398-4098-a0a3-ce2389512e3e service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:00.281602 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5be4b6-cbb5-4787-9e36-c61f9983b74e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:00.282301 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5be4b6-cbb5-4787-9e36-c61f9983b74e admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:00.282580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5be4b6-cbb5-4787-9e36-c61f9983b74e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:00.282939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5be4b6-cbb5-4787-9e36-c61f9983b74e admin admin] PATH_INFO: `/v3/users/7c30ae6f2a5d4a17a92fba80e598569c` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:00.284228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5be4b6-cbb5-4787-9e36-c61f9983b74e admin admin] RBAC: Authorizing `identity:delete_user(user_id=7c30ae6f2a5d4a17a92fba80e598569c)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:00.286176 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5be4b6-cbb5-4787-9e36-c61f9983b74e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:00.335576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7b5be4b6-cbb5-4787-9e36-c61f9983b74e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7c30ae6f2a5d4a17a92fba80e598569c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:00.336155 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7b5be4b6-cbb5-4787-9e36-c61f9983b74e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7c30ae6f2a5d4a17a92fba80e598569c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:00.348514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7b5be4b6-cbb5-4787-9e36-c61f9983b74e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7c30ae6f2a5d4a17a92fba80e598569c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:00.356546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7b5be4b6-cbb5-4787-9e36-c61f9983b74e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7c30ae6f2a5d4a17a92fba80e598569c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:00.363935 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7b5be4b6-cbb5-4787-9e36-c61f9983b74e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7c30ae6f2a5d4a17a92fba80e598569c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:00.370773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1141/2245] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:03:00 2022] DELETE /identity/v3/users/7c30ae6f2a5d4a17a92fba80e598569c => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:00.659033 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:00.660619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:00.674013 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcf1215-dc43-4123-bbe1-3b3a9fe0d4fb admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:00.734034 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12ae0fc-8da0-4887-8319-97439577d74d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:00.734665 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ae0fc-8da0-4887-8319-97439577d74d admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:00.734947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ae0fc-8da0-4887-8319-97439577d74d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:00.735220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ae0fc-8da0-4887-8319-97439577d74d admin admin] PATH_INFO: `/v3/projects/9f1e83d9daa7453fb8610b3f058a2b3e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:00.736384 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12ae0fc-8da0-4887-8319-97439577d74d admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f1e83d9daa7453fb8610b3f058a2b3e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:00.738283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12ae0fc-8da0-4887-8319-97439577d74d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:00.754830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:00.787597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d12ae0fc-8da0-4887-8319-97439577d74d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f1e83d9daa7453fb8610b3f058a2b3e', 'request_id': 'req-d12ae0fc-8da0-4887-8319-97439577d74d'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:00.795200 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d12ae0fc-8da0-4887-8319-97439577d74d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f1e83d9daa7453fb8610b3f058a2b3e', 'request_id': 'req-d12ae0fc-8da0-4887-8319-97439577d74d'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:00.800844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d12ae0fc-8da0-4887-8319-97439577d74d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f1e83d9daa7453fb8610b3f058a2b3e', 'request_id': 'req-d12ae0fc-8da0-4887-8319-97439577d74d'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:00.812739 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d12ae0fc-8da0-4887-8319-97439577d74d admin admin] The token cache is being invalidate because project 9f1e83d9daa7453fb8610b3f058a2b3e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:03:00.813029 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d12ae0fc-8da0-4887-8319-97439577d74d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:00.813671 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d12ae0fc-8da0-4887-8319-97439577d74d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:00.815147 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1105/2246] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:03:00 2022] DELETE /identity/v3/projects/9f1e83d9daa7453fb8610b3f058a2b3e => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:00.819997 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d093979-75ca-404b-a24f-3a0b2a086522 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:00.820468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d093979-75ca-404b-a24f-3a0b2a086522 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:00.820557 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d093979-75ca-404b-a24f-3a0b2a086522 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:00.841357 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-8d093979-75ca-404b-a24f-3a0b2a086522 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:00.842564 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8d093979-75ca-404b-a24f-3a0b2a086522 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:00.902431 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1142/2247] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:00.910266 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386c6ceb-26f1-483b-8fa3-0b840700f1dd None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:00.910550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386c6ceb-26f1-483b-8fa3-0b840700f1dd None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:00.910825 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386c6ceb-26f1-483b-8fa3-0b840700f1dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:00.931163 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-386c6ceb-26f1-483b-8fa3-0b840700f1dd None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:00.932383 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-386c6ceb-26f1-483b-8fa3-0b840700f1dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:00.940097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1106/2248] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:00.944137 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d093979-75ca-404b-a24f-3a0b2a086522 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:00.954433 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d093979-75ca-404b-a24f-3a0b2a086522 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:00.958941 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47265a2e-6276-4498-ab33-003fe673ea98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:00.959567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47265a2e-6276-4498-ab33-003fe673ea98 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:00.959856 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47265a2e-6276-4498-ab33-003fe673ea98 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:00.960155 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47265a2e-6276-4498-ab33-003fe673ea98 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:00.961288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47265a2e-6276-4498-ab33-003fe673ea98 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:03:00.961582 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47265a2e-6276-4498-ab33-003fe673ea98 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:00.963148 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47265a2e-6276-4498-ab33-003fe673ea98 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:00.974859 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1143/2249] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:03:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:03:00.979753 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386c6ceb-26f1-483b-8fa3-0b840700f1dd None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:00.988850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386c6ceb-26f1-483b-8fa3-0b840700f1dd None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:00.993589 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53fb99f-e98a-43f3-a01f-c8b16ed7ed58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:00.994182 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53fb99f-e98a-43f3-a01f-c8b16ed7ed58 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:00.994459 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53fb99f-e98a-43f3-a01f-c8b16ed7ed58 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:00.994791 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53fb99f-e98a-43f3-a01f-c8b16ed7ed58 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:00.995420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53fb99f-e98a-43f3-a01f-c8b16ed7ed58 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:00.997377 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53fb99f-e98a-43f3-a01f-c8b16ed7ed58 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:01.034677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1107/2250] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:03:00 2022] POST /identity/v3/projects => generated 375 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:01.038672 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47265a2e-6276-4498-ab33-003fe673ea98 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:01.050155 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47265a2e-6276-4498-ab33-003fe673ea98 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:01.053872 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:01.111246 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473d9073-63c8-4050-9380-ac066c2b5139 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:01.111867 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473d9073-63c8-4050-9380-ac066c2b5139 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:01.112142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473d9073-63c8-4050-9380-ac066c2b5139 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:01.112412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473d9073-63c8-4050-9380-ac066c2b5139 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:01.113033 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473d9073-63c8-4050-9380-ac066c2b5139 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:01.114929 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473d9073-63c8-4050-9380-ac066c2b5139 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:01.135634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1144/2251] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:03:01 2022] POST /identity/v3/users => generated 288 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:03:01.139882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53fb99f-e98a-43f3-a01f-c8b16ed7ed58 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:01.151174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53fb99f-e98a-43f3-a01f-c8b16ed7ed58 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:01.155883 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66382487-00d6-4460-88ab-ee138a15e033 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:01.156473 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66382487-00d6-4460-88ab-ee138a15e033 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:01.156751 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66382487-00d6-4460-88ab-ee138a15e033 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:01.157023 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66382487-00d6-4460-88ab-ee138a15e033 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:01.157664 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66382487-00d6-4460-88ab-ee138a15e033 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:01.159250 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66382487-00d6-4460-88ab-ee138a15e033 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:01.169920 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1108/2252] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:01 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:01.173758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473d9073-63c8-4050-9380-ac066c2b5139 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:01.184159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473d9073-63c8-4050-9380-ac066c2b5139 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:01.188712 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfcb9d2-620b-496f-8b49-8924aeffc908 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:01.189280 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfcb9d2-620b-496f-8b49-8924aeffc908 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:01.189559 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfcb9d2-620b-496f-8b49-8924aeffc908 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:01.189832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfcb9d2-620b-496f-8b49-8924aeffc908 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:01.190464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfcb9d2-620b-496f-8b49-8924aeffc908 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:01.192080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfcb9d2-620b-496f-8b49-8924aeffc908 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:01.203371 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1145/2253] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:01 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:01.207539 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66382487-00d6-4460-88ab-ee138a15e033 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:01.218016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66382487-00d6-4460-88ab-ee138a15e033 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:01.222653 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06842c4-8404-43eb-98cc-4ebc6f47f24c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:01.223291 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06842c4-8404-43eb-98cc-4ebc6f47f24c admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:01.223601 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06842c4-8404-43eb-98cc-4ebc6f47f24c admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:01.223884 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06842c4-8404-43eb-98cc-4ebc6f47f24c admin admin] PATH_INFO: `/v3/projects/a9ffa5a2cce44948a3c74936f1258538/users/9479617e530146bca0ba4c3aa9ac415c/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:01.238098 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06842c4-8404-43eb-98cc-4ebc6f47f24c admin admin] RBAC: Authorizing `identity:create_grant(project_id=a9ffa5a2cce44948a3c74936f1258538, user_id=9479617e530146bca0ba4c3aa9ac415c, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:01.240768 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06842c4-8404-43eb-98cc-4ebc6f47f24c admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:01.253373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1109/2254] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:03:01 2022] PUT /identity/v3/projects/a9ffa5a2cce44948a3c74936f1258538/users/9479617e530146bca0ba4c3aa9ac415c/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:01.265245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b962b0b0-f036-4b08-9b86-620c0b6f530f None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:01.265528 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b962b0b0-f036-4b08-9b86-620c0b6f530f None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:01.265802 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b962b0b0-f036-4b08-9b86-620c0b6f530f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:01.295661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-b962b0b0-f036-4b08-9b86-620c0b6f530f None None] MFA Rules not processed for user `9479617e530146bca0ba4c3aa9ac415c`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:01.296910 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b962b0b0-f036-4b08-9b86-620c0b6f530f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:01.363854 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1146/2255] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:01 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:01.368426 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06842c4-8404-43eb-98cc-4ebc6f47f24c admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:01.429159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06842c4-8404-43eb-98cc-4ebc6f47f24c admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:01.434261 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92108b8d-69f9-44ea-87fe-eef818b9c8af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:01.434889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92108b8d-69f9-44ea-87fe-eef818b9c8af admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:01.435168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92108b8d-69f9-44ea-87fe-eef818b9c8af admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:01.435436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92108b8d-69f9-44ea-87fe-eef818b9c8af admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:01.436144 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92108b8d-69f9-44ea-87fe-eef818b9c8af admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:01.438228 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92108b8d-69f9-44ea-87fe-eef818b9c8af admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:01.467151 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1110/2256] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:03:01 2022] POST /identity/v3/projects => generated 375 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:01.471222 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b962b0b0-f036-4b08-9b86-620c0b6f530f None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:01.482451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b962b0b0-f036-4b08-9b86-620c0b6f530f None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:01.548518 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2a7c3e-14b3-4dfc-a5b6-186269d38cfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:01.549131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2a7c3e-14b3-4dfc-a5b6-186269d38cfe admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:01.549410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2a7c3e-14b3-4dfc-a5b6-186269d38cfe admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:01.549684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2a7c3e-14b3-4dfc-a5b6-186269d38cfe admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:01.550320 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2a7c3e-14b3-4dfc-a5b6-186269d38cfe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:01.552358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2a7c3e-14b3-4dfc-a5b6-186269d38cfe admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:01.575647 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1147/2257] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:03:01 2022] POST /identity/v3/users => generated 294 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:01.578627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92108b8d-69f9-44ea-87fe-eef818b9c8af admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:01.588948 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92108b8d-69f9-44ea-87fe-eef818b9c8af admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:01.593486 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4039f0-092e-4fe3-81c2-8985f72a0860 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:01.594072 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4039f0-092e-4fe3-81c2-8985f72a0860 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:01.594347 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4039f0-092e-4fe3-81c2-8985f72a0860 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:01.594631 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4039f0-092e-4fe3-81c2-8985f72a0860 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:01.595252 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4039f0-092e-4fe3-81c2-8985f72a0860 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:01.596885 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4039f0-092e-4fe3-81c2-8985f72a0860 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:01.607237 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1111/2258] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:01 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:01.611351 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2a7c3e-14b3-4dfc-a5b6-186269d38cfe admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:01.618314 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:01.622037 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2a7c3e-14b3-4dfc-a5b6-186269d38cfe admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:01.627715 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c17f7c-b9ab-4302-8afb-2e531ffe9981 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:01.628381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c17f7c-b9ab-4302-8afb-2e531ffe9981 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:01.628662 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c17f7c-b9ab-4302-8afb-2e531ffe9981 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:01.628940 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c17f7c-b9ab-4302-8afb-2e531ffe9981 admin admin] PATH_INFO: `/v3/projects/b4a2bbef807948b883b7f2ee2686898e/users/dc353eecd71d4dcb97c9a5e32ace2cbf/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:01.644216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c17f7c-b9ab-4302-8afb-2e531ffe9981 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b4a2bbef807948b883b7f2ee2686898e, user_id=dc353eecd71d4dcb97c9a5e32ace2cbf, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:01.646895 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c17f7c-b9ab-4302-8afb-2e531ffe9981 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:01.663819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1148/2259] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:03:01 2022] PUT /identity/v3/projects/b4a2bbef807948b883b7f2ee2686898e/users/dc353eecd71d4dcb97c9a5e32ace2cbf/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:01.667879 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4039f0-092e-4fe3-81c2-8985f72a0860 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:01.677831 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4039f0-092e-4fe3-81c2-8985f72a0860 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:01.689695 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:01.735693 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fdb130-f6de-4082-ad7a-4a999ee9fa76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:01.736295 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fdb130-f6de-4082-ad7a-4a999ee9fa76 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:01.736593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fdb130-f6de-4082-ad7a-4a999ee9fa76 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:01.736866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fdb130-f6de-4082-ad7a-4a999ee9fa76 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:01.737525 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4fdb130-f6de-4082-ad7a-4a999ee9fa76 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:01.739111 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4fdb130-f6de-4082-ad7a-4a999ee9fa76 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:01.749785 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1112/2260] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:01 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:01.753697 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c17f7c-b9ab-4302-8afb-2e531ffe9981 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:01.764440 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c17f7c-b9ab-4302-8afb-2e531ffe9981 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:01.768989 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da01b364-5f2f-4e5a-8564-5dc1220db8c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:01.769627 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da01b364-5f2f-4e5a-8564-5dc1220db8c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:01.770025 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da01b364-5f2f-4e5a-8564-5dc1220db8c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:01.770303 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da01b364-5f2f-4e5a-8564-5dc1220db8c6 admin admin] PATH_INFO: `/v3/projects/b4a2bbef807948b883b7f2ee2686898e/users/dc353eecd71d4dcb97c9a5e32ace2cbf/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:01.772204 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da01b364-5f2f-4e5a-8564-5dc1220db8c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b4a2bbef807948b883b7f2ee2686898e, user_id=dc353eecd71d4dcb97c9a5e32ace2cbf, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:01.774814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da01b364-5f2f-4e5a-8564-5dc1220db8c6 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:01.784804 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1149/2261] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:03:01 2022] PUT /identity/v3/projects/b4a2bbef807948b883b7f2ee2686898e/users/dc353eecd71d4dcb97c9a5e32ace2cbf/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:01.795124 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36102cc-e7fe-4d32-a3a0-d1ca8cdacd8f None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:01.795417 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36102cc-e7fe-4d32-a3a0-d1ca8cdacd8f None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:01.795758 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36102cc-e7fe-4d32-a3a0-d1ca8cdacd8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:01.824891 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-c36102cc-e7fe-4d32-a3a0-d1ca8cdacd8f None None] MFA Rules not processed for user `dc353eecd71d4dcb97c9a5e32ace2cbf`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:01.826097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-c36102cc-e7fe-4d32-a3a0-d1ca8cdacd8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:01.893550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1113/2262] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:01 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:03:01.904094 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da01b364-5f2f-4e5a-8564-5dc1220db8c6 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:01.905811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-da01b364-5f2f-4e5a-8564-5dc1220db8c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:01.962966 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da01b364-5f2f-4e5a-8564-5dc1220db8c6 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:01.966569 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300d30a2-0bdf-4e34-ae8a-909ec452ab1d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:01.967159 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300d30a2-0bdf-4e34-ae8a-909ec452ab1d service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:01.967436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300d30a2-0bdf-4e34-ae8a-909ec452ab1d service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:01.967741 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300d30a2-0bdf-4e34-ae8a-909ec452ab1d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:02.023825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300d30a2-0bdf-4e34-ae8a-909ec452ab1d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:02.025256 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300d30a2-0bdf-4e34-ae8a-909ec452ab1d service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:02.034013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1150/2263] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:01 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:03:04.587772 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36102cc-e7fe-4d32-a3a0-d1ca8cdacd8f None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:04.654565 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36102cc-e7fe-4d32-a3a0-d1ca8cdacd8f None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:04.661266 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073c941d-1625-4849-b59d-97ce0fa99d51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:04.661906 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073c941d-1625-4849-b59d-97ce0fa99d51 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:04.662184 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073c941d-1625-4849-b59d-97ce0fa99d51 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:04.662457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073c941d-1625-4849-b59d-97ce0fa99d51 admin admin] PATH_INFO: `/v3/users/9479617e530146bca0ba4c3aa9ac415c` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:04.664180 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073c941d-1625-4849-b59d-97ce0fa99d51 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9479617e530146bca0ba4c3aa9ac415c)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:04.666191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073c941d-1625-4849-b59d-97ce0fa99d51 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:04.723607 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:04.732208 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-073c941d-1625-4849-b59d-97ce0fa99d51 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9479617e530146bca0ba4c3aa9ac415c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:04.739944 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-073c941d-1625-4849-b59d-97ce0fa99d51 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9479617e530146bca0ba4c3aa9ac415c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:04.740610 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-073c941d-1625-4849-b59d-97ce0fa99d51 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9479617e530146bca0ba4c3aa9ac415c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:04.741210 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-073c941d-1625-4849-b59d-97ce0fa99d51 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9479617e530146bca0ba4c3aa9ac415c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:04.754135 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-073c941d-1625-4849-b59d-97ce0fa99d51 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9479617e530146bca0ba4c3aa9ac415c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:04.768687 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1114/2264] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:03:04 2022] DELETE /identity/v3/users/9479617e530146bca0ba4c3aa9ac415c => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:04.773301 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4d1390-1ca1-4956-841b-abf4671a894b None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:04.773762 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4d1390-1ca1-4956-841b-abf4671a894b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:04.773854 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4d1390-1ca1-4956-841b-abf4671a894b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:04.792703 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-7e4d1390-1ca1-4956-841b-abf4671a894b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:04.793821 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7e4d1390-1ca1-4956-841b-abf4671a894b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:04.852302 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1151/2265] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:04.861964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073c941d-1625-4849-b59d-97ce0fa99d51 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:04.863393 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-073c941d-1625-4849-b59d-97ce0fa99d51 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:04.920858 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073c941d-1625-4849-b59d-97ce0fa99d51 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:04.924467 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926294aa-3a49-4f4c-9fa9-a7c2a41b84da service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:04.925060 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926294aa-3a49-4f4c-9fa9-a7c2a41b84da service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:04.925336 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926294aa-3a49-4f4c-9fa9-a7c2a41b84da service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:04.925600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926294aa-3a49-4f4c-9fa9-a7c2a41b84da service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:04.935042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926294aa-3a49-4f4c-9fa9-a7c2a41b84da service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:04.936516 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926294aa-3a49-4f4c-9fa9-a7c2a41b84da service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:04.945982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1115/2266] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:03:05.207581 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4d1390-1ca1-4956-841b-abf4671a894b None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:05.209554 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7e4d1390-1ca1-4956-841b-abf4671a894b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:05.228966 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4d1390-1ca1-4956-841b-abf4671a894b None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:05.235359 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad50e644-435d-4399-b9d4-cb7e44a6259f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:05.236050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad50e644-435d-4399-b9d4-cb7e44a6259f admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:05.236363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad50e644-435d-4399-b9d4-cb7e44a6259f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:05.236590 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad50e644-435d-4399-b9d4-cb7e44a6259f admin admin] PATH_INFO: `/v3/projects/a9ffa5a2cce44948a3c74936f1258538` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:05.237789 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad50e644-435d-4399-b9d4-cb7e44a6259f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a9ffa5a2cce44948a3c74936f1258538)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:05.239858 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad50e644-435d-4399-b9d4-cb7e44a6259f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:05.266038 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:05.301478 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ad50e644-435d-4399-b9d4-cb7e44a6259f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a9ffa5a2cce44948a3c74936f1258538', 'request_id': 'req-ad50e644-435d-4399-b9d4-cb7e44a6259f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:05.307605 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ad50e644-435d-4399-b9d4-cb7e44a6259f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a9ffa5a2cce44948a3c74936f1258538', 'request_id': 'req-ad50e644-435d-4399-b9d4-cb7e44a6259f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:05.314577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ad50e644-435d-4399-b9d4-cb7e44a6259f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a9ffa5a2cce44948a3c74936f1258538', 'request_id': 'req-ad50e644-435d-4399-b9d4-cb7e44a6259f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:05.327763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ad50e644-435d-4399-b9d4-cb7e44a6259f admin admin] The token cache is being invalidate because project a9ffa5a2cce44948a3c74936f1258538 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:03:05.328054 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ad50e644-435d-4399-b9d4-cb7e44a6259f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:05.328665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-ad50e644-435d-4399-b9d4-cb7e44a6259f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:05.330228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1152/2267] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:03:05 2022] DELETE /identity/v3/projects/a9ffa5a2cce44948a3c74936f1258538 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:05.334325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926294aa-3a49-4f4c-9fa9-a7c2a41b84da service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:05.344794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926294aa-3a49-4f4c-9fa9-a7c2a41b84da service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:05.400316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fdc8d0-14a9-4b29-9889-dab5f71616e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:05.400928 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fdc8d0-14a9-4b29-9889-dab5f71616e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:05.401204 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fdc8d0-14a9-4b29-9889-dab5f71616e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:05.401475 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fdc8d0-14a9-4b29-9889-dab5f71616e7 admin admin] PATH_INFO: `/v3/users/dc353eecd71d4dcb97c9a5e32ace2cbf` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:05.402580 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fdc8d0-14a9-4b29-9889-dab5f71616e7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dc353eecd71d4dcb97c9a5e32ace2cbf)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:05.404522 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fdc8d0-14a9-4b29-9889-dab5f71616e7 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:05.450357 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-21fdc8d0-14a9-4b29-9889-dab5f71616e7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dc353eecd71d4dcb97c9a5e32ace2cbf'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:05.457318 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-21fdc8d0-14a9-4b29-9889-dab5f71616e7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dc353eecd71d4dcb97c9a5e32ace2cbf'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:05.457978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-21fdc8d0-14a9-4b29-9889-dab5f71616e7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dc353eecd71d4dcb97c9a5e32ace2cbf'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:05.458596 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-21fdc8d0-14a9-4b29-9889-dab5f71616e7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dc353eecd71d4dcb97c9a5e32ace2cbf'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:05.463610 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:05.473404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-21fdc8d0-14a9-4b29-9889-dab5f71616e7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dc353eecd71d4dcb97c9a5e32ace2cbf'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:05.486983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1116/2268] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:03:05 2022] DELETE /identity/v3/users/dc353eecd71d4dcb97c9a5e32ace2cbf => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:05.760304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad50e644-435d-4399-b9d4-cb7e44a6259f admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:05.761879 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-ad50e644-435d-4399-b9d4-cb7e44a6259f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:05.776459 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad50e644-435d-4399-b9d4-cb7e44a6259f admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:05.840273 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75c424a-9e0b-476c-a9d3-4de498a732bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:05.840938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75c424a-9e0b-476c-a9d3-4de498a732bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:05.841216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75c424a-9e0b-476c-a9d3-4de498a732bb admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:05.841485 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75c424a-9e0b-476c-a9d3-4de498a732bb admin admin] PATH_INFO: `/v3/projects/b4a2bbef807948b883b7f2ee2686898e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:05.842637 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75c424a-9e0b-476c-a9d3-4de498a732bb admin admin] RBAC: Authorizing `identity:delete_project(project_id=b4a2bbef807948b883b7f2ee2686898e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:05.844586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75c424a-9e0b-476c-a9d3-4de498a732bb admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:05.894601 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c75c424a-9e0b-476c-a9d3-4de498a732bb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b4a2bbef807948b883b7f2ee2686898e', 'request_id': 'req-c75c424a-9e0b-476c-a9d3-4de498a732bb'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:05.900512 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c75c424a-9e0b-476c-a9d3-4de498a732bb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b4a2bbef807948b883b7f2ee2686898e', 'request_id': 'req-c75c424a-9e0b-476c-a9d3-4de498a732bb'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:05.906609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c75c424a-9e0b-476c-a9d3-4de498a732bb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b4a2bbef807948b883b7f2ee2686898e', 'request_id': 'req-c75c424a-9e0b-476c-a9d3-4de498a732bb'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:05.921789 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c75c424a-9e0b-476c-a9d3-4de498a732bb admin admin] The token cache is being invalidate because project b4a2bbef807948b883b7f2ee2686898e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:03:05.922075 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c75c424a-9e0b-476c-a9d3-4de498a732bb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:05.922724 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c75c424a-9e0b-476c-a9d3-4de498a732bb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:05.924282 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1153/2269] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:03:05 2022] DELETE /identity/v3/projects/b4a2bbef807948b883b7f2ee2686898e => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:05.929145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d75d565-f8de-4dca-a3d6-ffba368ddd3e None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:05.929808 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d75d565-f8de-4dca-a3d6-ffba368ddd3e None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:05.930240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d75d565-f8de-4dca-a3d6-ffba368ddd3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:05.950189 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-8d75d565-f8de-4dca-a3d6-ffba368ddd3e None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:05.951384 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-8d75d565-f8de-4dca-a3d6-ffba368ddd3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:06.013080 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1117/2270] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:06.020789 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d743545-0f31-4543-91bb-b43a8e818bc2 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:06.021065 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d743545-0f31-4543-91bb-b43a8e818bc2 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:06.021577 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d743545-0f31-4543-91bb-b43a8e818bc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:06.041591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-1d743545-0f31-4543-91bb-b43a8e818bc2 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:06.042781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1d743545-0f31-4543-91bb-b43a8e818bc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:06.050582 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1154/2271] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:06.056114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d75d565-f8de-4dca-a3d6-ffba368ddd3e None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:06.066217 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d75d565-f8de-4dca-a3d6-ffba368ddd3e None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:06.072615 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e356d90f-f86e-4f0c-a8af-7bdec8e2f6b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:06.073228 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e356d90f-f86e-4f0c-a8af-7bdec8e2f6b8 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:06.073512 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e356d90f-f86e-4f0c-a8af-7bdec8e2f6b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:06.073787 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e356d90f-f86e-4f0c-a8af-7bdec8e2f6b8 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:06.074445 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e356d90f-f86e-4f0c-a8af-7bdec8e2f6b8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:03:06.074752 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e356d90f-f86e-4f0c-a8af-7bdec8e2f6b8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:06.076437 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e356d90f-f86e-4f0c-a8af-7bdec8e2f6b8 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:06.089074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1118/2272] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:03:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:03:06.093458 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d743545-0f31-4543-91bb-b43a8e818bc2 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:06.102823 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d743545-0f31-4543-91bb-b43a8e818bc2 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:06.107421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f89ac54-2bef-4d09-bc5c-92174891c2f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:06.108047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f89ac54-2bef-4d09-bc5c-92174891c2f2 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:06.108325 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f89ac54-2bef-4d09-bc5c-92174891c2f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:06.108595 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f89ac54-2bef-4d09-bc5c-92174891c2f2 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:06.109262 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f89ac54-2bef-4d09-bc5c-92174891c2f2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:06.111136 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f89ac54-2bef-4d09-bc5c-92174891c2f2 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:06.140163 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1155/2273] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:03:06 2022] POST /identity/v3/projects => generated 371 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:06.144486 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e356d90f-f86e-4f0c-a8af-7bdec8e2f6b8 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:06.155970 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e356d90f-f86e-4f0c-a8af-7bdec8e2f6b8 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:06.230306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b31d5c1-a43a-4039-abbb-3e680264a79d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:06.230912 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b31d5c1-a43a-4039-abbb-3e680264a79d admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:06.231192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b31d5c1-a43a-4039-abbb-3e680264a79d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:06.231466 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b31d5c1-a43a-4039-abbb-3e680264a79d admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:06.232168 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b31d5c1-a43a-4039-abbb-3e680264a79d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:06.234000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b31d5c1-a43a-4039-abbb-3e680264a79d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:06.256385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1119/2274] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:03:06 2022] POST /identity/v3/users => generated 286 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:06.261928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f89ac54-2bef-4d09-bc5c-92174891c2f2 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:06.274533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f89ac54-2bef-4d09-bc5c-92174891c2f2 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:06.279389 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3563b77c-ff48-4c72-9fed-60d68e6c9916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:06.280017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3563b77c-ff48-4c72-9fed-60d68e6c9916 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:06.280295 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3563b77c-ff48-4c72-9fed-60d68e6c9916 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:06.280562 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3563b77c-ff48-4c72-9fed-60d68e6c9916 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:06.281199 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3563b77c-ff48-4c72-9fed-60d68e6c9916 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:06.282775 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3563b77c-ff48-4c72-9fed-60d68e6c9916 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:06.284154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:06.295704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1156/2275] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:06 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:06.299469 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b31d5c1-a43a-4039-abbb-3e680264a79d admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:06.319241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b31d5c1-a43a-4039-abbb-3e680264a79d admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:06.325342 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7672e4c-204d-463e-9aaf-ee10af4d735e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:06.326010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7672e4c-204d-463e-9aaf-ee10af4d735e admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:06.326469 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7672e4c-204d-463e-9aaf-ee10af4d735e admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:06.326805 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7672e4c-204d-463e-9aaf-ee10af4d735e admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:06.327596 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7672e4c-204d-463e-9aaf-ee10af4d735e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:06.329315 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7672e4c-204d-463e-9aaf-ee10af4d735e admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:06.340509 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1120/2276] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:06 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:06.344544 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3563b77c-ff48-4c72-9fed-60d68e6c9916 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:06.354898 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3563b77c-ff48-4c72-9fed-60d68e6c9916 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:06.359601 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4badd939-123d-4b4a-a1e9-f9eed569a78f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:06.360346 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4badd939-123d-4b4a-a1e9-f9eed569a78f admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:06.360634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4badd939-123d-4b4a-a1e9-f9eed569a78f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:06.360911 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4badd939-123d-4b4a-a1e9-f9eed569a78f admin admin] PATH_INFO: `/v3/projects/e774aa6b8fe944c5a408a2910caafd7f/users/f63fcc3f3e8246819c25f4c8a66e0d22/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:06.376441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4badd939-123d-4b4a-a1e9-f9eed569a78f admin admin] RBAC: Authorizing `identity:create_grant(project_id=e774aa6b8fe944c5a408a2910caafd7f, user_id=f63fcc3f3e8246819c25f4c8a66e0d22, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:06.379086 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4badd939-123d-4b4a-a1e9-f9eed569a78f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:06.394187 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1157/2277] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:03:06 2022] PUT /identity/v3/projects/e774aa6b8fe944c5a408a2910caafd7f/users/f63fcc3f3e8246819c25f4c8a66e0d22/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:06.469623 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f07cdb-a43c-44c8-b160-1156a119ada0 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:06.470131 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f07cdb-a43c-44c8-b160-1156a119ada0 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:06.470283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f07cdb-a43c-44c8-b160-1156a119ada0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:06.499156 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-84f07cdb-a43c-44c8-b160-1156a119ada0 None None] MFA Rules not processed for user `f63fcc3f3e8246819c25f4c8a66e0d22`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:06.500669 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-84f07cdb-a43c-44c8-b160-1156a119ada0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:06.541795 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:DELETE=3,keystone:SELECT=93,keystone:INSERT=4 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:06.579782 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1121/2278] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:06 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:06.593932 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4badd939-123d-4b4a-a1e9-f9eed569a78f admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:06.595665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-4badd939-123d-4b4a-a1e9-f9eed569a78f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:06.685568 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4badd939-123d-4b4a-a1e9-f9eed569a78f admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:06.689763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa77742d-8754-45ea-bb89-4361814e3c35 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9808dcba0b64027bb70e31d1569957d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:06.690375 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa77742d-8754-45ea-bb89-4361814e3c35 service glance] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:06.690674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa77742d-8754-45ea-bb89-4361814e3c35 service glance] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:06.690961 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa77742d-8754-45ea-bb89-4361814e3c35 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:06.700541 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa77742d-8754-45ea-bb89-4361814e3c35 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:06.701970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa77742d-8754-45ea-bb89-4361814e3c35 service glance] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:06.711536 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1158/2279] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:06 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:03:16.576848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=12 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:16.706723 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=38,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:21.019682 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f07cdb-a43c-44c8-b160-1156a119ada0 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:21.021150 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-84f07cdb-a43c-44c8-b160-1156a119ada0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:21.081016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f07cdb-a43c-44c8-b160-1156a119ada0 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:21.084695 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8828a27c-09c3-4cf6-b63d-e55d9bea6b58 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:21.085871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8828a27c-09c3-4cf6-b63d-e55d9bea6b58 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:21.086194 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8828a27c-09c3-4cf6-b63d-e55d9bea6b58 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:21.086465 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8828a27c-09c3-4cf6-b63d-e55d9bea6b58 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:21.088246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-8828a27c-09c3-4cf6-b63d-e55d9bea6b58 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:21.151070 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8828a27c-09c3-4cf6-b63d-e55d9bea6b58 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:21.152538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8828a27c-09c3-4cf6-b63d-e55d9bea6b58 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:21.162793 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1122/2280] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:03:22.611417 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa77742d-8754-45ea-bb89-4361814e3c35 service glance] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:22.613139 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-aa77742d-8754-45ea-bb89-4361814e3c35 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:22.635039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa77742d-8754-45ea-bb89-4361814e3c35 service glance] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:22.639518 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76adf6c-fb7e-4409-991e-a8fe693b0094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:22.640198 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76adf6c-fb7e-4409-991e-a8fe693b0094 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:22.640490 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76adf6c-fb7e-4409-991e-a8fe693b0094 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:22.640760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76adf6c-fb7e-4409-991e-a8fe693b0094 admin admin] PATH_INFO: `/v3/users/c8c9e3dd67a4491eb7ab10a5a7e8af09` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:22.642158 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76adf6c-fb7e-4409-991e-a8fe693b0094 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c8c9e3dd67a4491eb7ab10a5a7e8af09)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:22.644377 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76adf6c-fb7e-4409-991e-a8fe693b0094 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:22.708173 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a76adf6c-fb7e-4409-991e-a8fe693b0094 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c8c9e3dd67a4491eb7ab10a5a7e8af09'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:22.709130 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a76adf6c-fb7e-4409-991e-a8fe693b0094 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c8c9e3dd67a4491eb7ab10a5a7e8af09'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:22.733139 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a76adf6c-fb7e-4409-991e-a8fe693b0094 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c8c9e3dd67a4491eb7ab10a5a7e8af09'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:22.743399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a76adf6c-fb7e-4409-991e-a8fe693b0094 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c8c9e3dd67a4491eb7ab10a5a7e8af09'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:22.753850 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-a76adf6c-fb7e-4409-991e-a8fe693b0094 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c8c9e3dd67a4491eb7ab10a5a7e8af09'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:22.766621 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1159/2281] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:03:22 2022] DELETE /identity/v3/users/c8c9e3dd67a4491eb7ab10a5a7e8af09 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:22.768150 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff3cb82-0ea7-45f4-a078-a6e7b1151a6f None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:22.768642 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff3cb82-0ea7-45f4-a078-a6e7b1151a6f None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:22.770525 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff3cb82-0ea7-45f4-a078-a6e7b1151a6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:22.806996 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-2ff3cb82-0ea7-45f4-a078-a6e7b1151a6f None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:22.808767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2ff3cb82-0ea7-45f4-a078-a6e7b1151a6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:22.891705 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1123/2282] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:22.903173 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76adf6c-fb7e-4409-991e-a8fe693b0094 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:22.905031 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a76adf6c-fb7e-4409-991e-a8fe693b0094 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:22.989675 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76adf6c-fb7e-4409-991e-a8fe693b0094 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:22.997950 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfba544-c5d4-4cc4-934e-c86efe67da09 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:22.998825 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfba544-c5d4-4cc4-934e-c86efe67da09 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:22.999283 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfba544-c5d4-4cc4-934e-c86efe67da09 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:22.999738 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfba544-c5d4-4cc4-934e-c86efe67da09 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:23.025938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfba544-c5d4-4cc4-934e-c86efe67da09 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:23.027404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfba544-c5d4-4cc4-934e-c86efe67da09 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:23.049549 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1160/2283] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:03:23.771471 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff3cb82-0ea7-45f4-a078-a6e7b1151a6f None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:23.775119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2ff3cb82-0ea7-45f4-a078-a6e7b1151a6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:23.799178 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff3cb82-0ea7-45f4-a078-a6e7b1151a6f None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:23.806527 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1aefdb-be41-4fa1-bd72-8db2166c24d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:23.807267 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1aefdb-be41-4fa1-bd72-8db2166c24d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:23.807825 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1aefdb-be41-4fa1-bd72-8db2166c24d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:23.808391 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1aefdb-be41-4fa1-bd72-8db2166c24d7 admin admin] PATH_INFO: `/v3/projects/6144ecc8e2bb4918af3b43abaf431eb0` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:23.809790 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1aefdb-be41-4fa1-bd72-8db2166c24d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6144ecc8e2bb4918af3b43abaf431eb0)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:23.812240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1aefdb-be41-4fa1-bd72-8db2166c24d7 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:23.888202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:23.892088 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-db1aefdb-be41-4fa1-bd72-8db2166c24d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6144ecc8e2bb4918af3b43abaf431eb0', 'request_id': 'req-db1aefdb-be41-4fa1-bd72-8db2166c24d7'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:23.901993 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-db1aefdb-be41-4fa1-bd72-8db2166c24d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6144ecc8e2bb4918af3b43abaf431eb0', 'request_id': 'req-db1aefdb-be41-4fa1-bd72-8db2166c24d7'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:23.908459 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-db1aefdb-be41-4fa1-bd72-8db2166c24d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6144ecc8e2bb4918af3b43abaf431eb0', 'request_id': 'req-db1aefdb-be41-4fa1-bd72-8db2166c24d7'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:23.920824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-db1aefdb-be41-4fa1-bd72-8db2166c24d7 admin admin] The token cache is being invalidate because project 6144ecc8e2bb4918af3b43abaf431eb0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:03:23.921119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-db1aefdb-be41-4fa1-bd72-8db2166c24d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:23.921800 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-db1aefdb-be41-4fa1-bd72-8db2166c24d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:23.923294 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1124/2284] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:03:23 2022] DELETE /identity/v3/projects/6144ecc8e2bb4918af3b43abaf431eb0 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:23.928534 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5832d594-62b3-4618-9be9-fd4bdfbac73b None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:23.929105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5832d594-62b3-4618-9be9-fd4bdfbac73b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:23.929528 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5832d594-62b3-4618-9be9-fd4bdfbac73b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:23.952174 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-5832d594-62b3-4618-9be9-fd4bdfbac73b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:23.953546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5832d594-62b3-4618-9be9-fd4bdfbac73b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:24.018039 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1161/2285] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:03:24.026783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09a8391-8bd2-4390-9166-06af6fd6f3c8 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:24.027070 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09a8391-8bd2-4390-9166-06af6fd6f3c8 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:24.027343 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09a8391-8bd2-4390-9166-06af6fd6f3c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:24.046785 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-f09a8391-8bd2-4390-9166-06af6fd6f3c8 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:24.048042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-f09a8391-8bd2-4390-9166-06af6fd6f3c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:24.058330 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1125/2286] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:24.062467 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5832d594-62b3-4618-9be9-fd4bdfbac73b None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:24.075202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5832d594-62b3-4618-9be9-fd4bdfbac73b None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:24.081030 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d12c08b-6bd4-4c1a-93b2-518dfad4a830 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:24.081669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d12c08b-6bd4-4c1a-93b2-518dfad4a830 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:24.081952 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d12c08b-6bd4-4c1a-93b2-518dfad4a830 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:24.082228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d12c08b-6bd4-4c1a-93b2-518dfad4a830 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:24.082885 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d12c08b-6bd4-4c1a-93b2-518dfad4a830 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:03:24.083194 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d12c08b-6bd4-4c1a-93b2-518dfad4a830 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:24.085134 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d12c08b-6bd4-4c1a-93b2-518dfad4a830 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:24.097017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1162/2287] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:03:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:03:24.103336 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09a8391-8bd2-4390-9166-06af6fd6f3c8 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:24.113160 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09a8391-8bd2-4390-9166-06af6fd6f3c8 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:24.119515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14522a3e-7c52-4369-b12b-119175befed9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:24.120574 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14522a3e-7c52-4369-b12b-119175befed9 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:24.120857 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14522a3e-7c52-4369-b12b-119175befed9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:24.121136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14522a3e-7c52-4369-b12b-119175befed9 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:24.121771 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14522a3e-7c52-4369-b12b-119175befed9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:24.123912 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14522a3e-7c52-4369-b12b-119175befed9 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:24.155162 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1126/2288] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:03:24 2022] POST /identity/v3/projects => generated 371 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:24.156293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d12c08b-6bd4-4c1a-93b2-518dfad4a830 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:24.161530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:24.168875 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d12c08b-6bd4-4c1a-93b2-518dfad4a830 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:24.238864 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccaa603-d1ac-45ae-bf38-74122ae324ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:24.239586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccaa603-d1ac-45ae-bf38-74122ae324ef admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:24.240012 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccaa603-d1ac-45ae-bf38-74122ae324ef admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:24.240364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccaa603-d1ac-45ae-bf38-74122ae324ef admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:24.242106 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccaa603-d1ac-45ae-bf38-74122ae324ef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:24.244762 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccaa603-d1ac-45ae-bf38-74122ae324ef admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:24.277945 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1163/2289] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:03:24 2022] POST /identity/v3/users => generated 286 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:03:24.283194 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14522a3e-7c52-4369-b12b-119175befed9 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:24.295175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14522a3e-7c52-4369-b12b-119175befed9 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:24.300081 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f7c317-e8aa-47e7-9167-3be7d6334af5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:24.300767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f7c317-e8aa-47e7-9167-3be7d6334af5 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:24.301121 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f7c317-e8aa-47e7-9167-3be7d6334af5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:24.301471 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f7c317-e8aa-47e7-9167-3be7d6334af5 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:24.302192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f7c317-e8aa-47e7-9167-3be7d6334af5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:24.305330 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f7c317-e8aa-47e7-9167-3be7d6334af5 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:24.320480 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1127/2290] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:24 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:24.324228 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccaa603-d1ac-45ae-bf38-74122ae324ef admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:24.335957 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccaa603-d1ac-45ae-bf38-74122ae324ef admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:24.341074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ccc369d-5413-4984-b2af-6d12dc3302d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:24.342391 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccc369d-5413-4984-b2af-6d12dc3302d5 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:24.342747 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccc369d-5413-4984-b2af-6d12dc3302d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:24.343027 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccc369d-5413-4984-b2af-6d12dc3302d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:24.343822 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ccc369d-5413-4984-b2af-6d12dc3302d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:24.345554 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ccc369d-5413-4984-b2af-6d12dc3302d5 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:24.359477 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1164/2291] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:24 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:24.363253 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f7c317-e8aa-47e7-9167-3be7d6334af5 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:24.389443 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f7c317-e8aa-47e7-9167-3be7d6334af5 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:24.397582 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511bdbdf-4528-45e9-85d8-798cdb0df9bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:24.398530 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511bdbdf-4528-45e9-85d8-798cdb0df9bd admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:24.398817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511bdbdf-4528-45e9-85d8-798cdb0df9bd admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:24.399100 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511bdbdf-4528-45e9-85d8-798cdb0df9bd admin admin] PATH_INFO: `/v3/projects/357cf607c19548859a14c92a4e90f148/users/c6dc53b3c9e041a7ae977b8f9c4fb478/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:24.423221 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511bdbdf-4528-45e9-85d8-798cdb0df9bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=357cf607c19548859a14c92a4e90f148, user_id=c6dc53b3c9e041a7ae977b8f9c4fb478, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:24.426151 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511bdbdf-4528-45e9-85d8-798cdb0df9bd admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:24.447654 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1128/2292] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:03:24 2022] PUT /identity/v3/projects/357cf607c19548859a14c92a4e90f148/users/c6dc53b3c9e041a7ae977b8f9c4fb478/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:24.449506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8ff662-fbc2-4dd5-873e-67de58aa613b None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:24.449819 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8ff662-fbc2-4dd5-873e-67de58aa613b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:24.450101 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8ff662-fbc2-4dd5-873e-67de58aa613b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:24.485860 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-9a8ff662-fbc2-4dd5-873e-67de58aa613b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:24.487339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-9a8ff662-fbc2-4dd5-873e-67de58aa613b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:24.580779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1165/2293] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:24.592539 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511bdbdf-4528-45e9-85d8-798cdb0df9bd admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:24.594425 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-511bdbdf-4528-45e9-85d8-798cdb0df9bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:24.707963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511bdbdf-4528-45e9-85d8-798cdb0df9bd admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:24.714326 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8902967-9669-47f7-8c17-a69ae4cac892 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:24.715135 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8902967-9669-47f7-8c17-a69ae4cac892 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:24.715684 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8902967-9669-47f7-8c17-a69ae4cac892 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:24.716135 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8902967-9669-47f7-8c17-a69ae4cac892 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:24.736017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8902967-9669-47f7-8c17-a69ae4cac892 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:24.738459 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8902967-9669-47f7-8c17-a69ae4cac892 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:24.764250 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1129/2294] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:03:24.956847 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8ff662-fbc2-4dd5-873e-67de58aa613b None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:24.959069 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-9a8ff662-fbc2-4dd5-873e-67de58aa613b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:25.042105 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8902967-9669-47f7-8c17-a69ae4cac892 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:25.043712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d8902967-9669-47f7-8c17-a69ae4cac892 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:25.070750 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8902967-9669-47f7-8c17-a69ae4cac892 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:25.076238 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdb4902-bc28-499c-ba25-b8c668cbc89f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:25.076879 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdb4902-bc28-499c-ba25-b8c668cbc89f admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:25.077055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8ff662-fbc2-4dd5-873e-67de58aa613b None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:25.077167 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdb4902-bc28-499c-ba25-b8c668cbc89f admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:25.077419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdb4902-bc28-499c-ba25-b8c668cbc89f admin admin] PATH_INFO: `/v3/users/cdcbcb2ed9e44dbe8444e531fc1eb707` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:25.079383 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdb4902-bc28-499c-ba25-b8c668cbc89f admin admin] RBAC: Authorizing `identity:delete_user(user_id=cdcbcb2ed9e44dbe8444e531fc1eb707)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:25.081523 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdb4902-bc28-499c-ba25-b8c668cbc89f admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:25.083579 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953b69e2-4380-4d75-a04a-a619319725d8 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70ea2aaf529b46dabb70b71f4dcc3860', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:25.084289 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953b69e2-4380-4d75-a04a-a619319725d8 service placement] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:25.084639 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953b69e2-4380-4d75-a04a-a619319725d8 service placement] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:25.085636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953b69e2-4380-4d75-a04a-a619319725d8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:25.087676 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-953b69e2-4380-4d75-a04a-a619319725d8 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:25.102948 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953b69e2-4380-4d75-a04a-a619319725d8 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:25.104857 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953b69e2-4380-4d75-a04a-a619319725d8 service placement] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:25.119399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1166/2295] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:03:25.142845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-3fdb4902-bc28-499c-ba25-b8c668cbc89f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cdcbcb2ed9e44dbe8444e531fc1eb707'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:25.151999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-3fdb4902-bc28-499c-ba25-b8c668cbc89f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cdcbcb2ed9e44dbe8444e531fc1eb707'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:25.153495 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-3fdb4902-bc28-499c-ba25-b8c668cbc89f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cdcbcb2ed9e44dbe8444e531fc1eb707'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:25.154306 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-3fdb4902-bc28-499c-ba25-b8c668cbc89f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cdcbcb2ed9e44dbe8444e531fc1eb707'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:25.167748 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:25.176953 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-3fdb4902-bc28-499c-ba25-b8c668cbc89f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cdcbcb2ed9e44dbe8444e531fc1eb707'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:25.199381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1130/2296] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:03:25 2022] DELETE /identity/v3/users/cdcbcb2ed9e44dbe8444e531fc1eb707 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:03:25.483521 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953b69e2-4380-4d75-a04a-a619319725d8 service placement] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:25.500811 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953b69e2-4380-4d75-a04a-a619319725d8 service placement] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:25.511028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:25.584256 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f799a8b8-0d3f-4bd9-a32b-9e87fb29de5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:25.584985 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f799a8b8-0d3f-4bd9-a32b-9e87fb29de5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:25.585336 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f799a8b8-0d3f-4bd9-a32b-9e87fb29de5f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:25.585539 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f799a8b8-0d3f-4bd9-a32b-9e87fb29de5f admin admin] PATH_INFO: `/v3/projects/f629aa044ca845069237ecb8d889284e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:25.586712 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f799a8b8-0d3f-4bd9-a32b-9e87fb29de5f admin admin] RBAC: Authorizing `identity:delete_project(project_id=f629aa044ca845069237ecb8d889284e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:25.588928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f799a8b8-0d3f-4bd9-a32b-9e87fb29de5f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:25.648000 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f799a8b8-0d3f-4bd9-a32b-9e87fb29de5f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f629aa044ca845069237ecb8d889284e', 'request_id': 'req-f799a8b8-0d3f-4bd9-a32b-9e87fb29de5f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:25.661943 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f799a8b8-0d3f-4bd9-a32b-9e87fb29de5f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f629aa044ca845069237ecb8d889284e', 'request_id': 'req-f799a8b8-0d3f-4bd9-a32b-9e87fb29de5f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:25.673166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f799a8b8-0d3f-4bd9-a32b-9e87fb29de5f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f629aa044ca845069237ecb8d889284e', 'request_id': 'req-f799a8b8-0d3f-4bd9-a32b-9e87fb29de5f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:25.690477 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f799a8b8-0d3f-4bd9-a32b-9e87fb29de5f admin admin] The token cache is being invalidate because project f629aa044ca845069237ecb8d889284e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:03:25.690849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f799a8b8-0d3f-4bd9-a32b-9e87fb29de5f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:25.691682 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f799a8b8-0d3f-4bd9-a32b-9e87fb29de5f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:25.697465 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1167/2297] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:03:25 2022] DELETE /identity/v3/projects/f629aa044ca845069237ecb8d889284e => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:03:25.699438 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81969ae2-7583-47cd-8331-fec08157cc8f None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:25.699753 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81969ae2-7583-47cd-8331-fec08157cc8f None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:25.700030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81969ae2-7583-47cd-8331-fec08157cc8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:25.724408 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-81969ae2-7583-47cd-8331-fec08157cc8f None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:25.725721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-81969ae2-7583-47cd-8331-fec08157cc8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:25.790725 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1131/2298] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:25.798876 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a505817-63f2-4cb4-9d51-229097743313 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:25.799167 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a505817-63f2-4cb4-9d51-229097743313 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:25.799416 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a505817-63f2-4cb4-9d51-229097743313 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:25.823585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-5a505817-63f2-4cb4-9d51-229097743313 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:25.825139 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5a505817-63f2-4cb4-9d51-229097743313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:25.836503 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1168/2299] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:25.840697 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81969ae2-7583-47cd-8331-fec08157cc8f None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:25.852197 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81969ae2-7583-47cd-8331-fec08157cc8f None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:25.857958 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47834e1-19b4-4403-8a53-f1bb26ae94f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:25.858719 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47834e1-19b4-4403-8a53-f1bb26ae94f8 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:25.859000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47834e1-19b4-4403-8a53-f1bb26ae94f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:25.859275 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47834e1-19b4-4403-8a53-f1bb26ae94f8 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:25.860068 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47834e1-19b4-4403-8a53-f1bb26ae94f8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:03:25.860373 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47834e1-19b4-4403-8a53-f1bb26ae94f8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:25.862116 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47834e1-19b4-4403-8a53-f1bb26ae94f8 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:25.884395 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1132/2300] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:03:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 15:03:25.889105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a505817-63f2-4cb4-9d51-229097743313 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:25.901140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a505817-63f2-4cb4-9d51-229097743313 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:25.907230 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8fbad78-82f0-42a7-af20-89216c3a379f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:25.907880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fbad78-82f0-42a7-af20-89216c3a379f admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:25.908163 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fbad78-82f0-42a7-af20-89216c3a379f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:25.908879 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fbad78-82f0-42a7-af20-89216c3a379f admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:25.909606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8fbad78-82f0-42a7-af20-89216c3a379f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:25.912273 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8fbad78-82f0-42a7-af20-89216c3a379f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:25.947580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1169/2301] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:03:25 2022] POST /identity/v3/projects => generated 377 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:25.948697 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47834e1-19b4-4403-8a53-f1bb26ae94f8 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:25.960064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47834e1-19b4-4403-8a53-f1bb26ae94f8 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:26.029432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fda580-ee43-4144-b4b0-c9a18aee04f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:26.030438 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fda580-ee43-4144-b4b0-c9a18aee04f6 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:26.030438 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fda580-ee43-4144-b4b0-c9a18aee04f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:26.030625 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fda580-ee43-4144-b4b0-c9a18aee04f6 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:26.031200 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fda580-ee43-4144-b4b0-c9a18aee04f6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:26.033186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fda580-ee43-4144-b4b0-c9a18aee04f6 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:26.069746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1133/2302] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:03:25 2022] POST /identity/v3/users => generated 289 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:03:26.070888 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8fbad78-82f0-42a7-af20-89216c3a379f admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:26.082989 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8fbad78-82f0-42a7-af20-89216c3a379f admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:26.088218 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca99a58-1849-49dc-99cd-48a75002fd14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:26.088996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca99a58-1849-49dc-99cd-48a75002fd14 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:26.089281 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca99a58-1849-49dc-99cd-48a75002fd14 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:26.089554 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca99a58-1849-49dc-99cd-48a75002fd14 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:26.090202 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca99a58-1849-49dc-99cd-48a75002fd14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:26.091849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca99a58-1849-49dc-99cd-48a75002fd14 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:26.105095 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1170/2303] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:26 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:03:26.109209 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fda580-ee43-4144-b4b0-c9a18aee04f6 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:26.129452 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fda580-ee43-4144-b4b0-c9a18aee04f6 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:26.139239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfea28b-09d6-49be-9a83-bf63ce1cc9ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:26.140191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfea28b-09d6-49be-9a83-bf63ce1cc9ff admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:26.140538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfea28b-09d6-49be-9a83-bf63ce1cc9ff admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:26.140821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfea28b-09d6-49be-9a83-bf63ce1cc9ff admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:26.141478 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfea28b-09d6-49be-9a83-bf63ce1cc9ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:26.143307 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfea28b-09d6-49be-9a83-bf63ce1cc9ff admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:26.159979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1134/2304] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:26 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:26.161341 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca99a58-1849-49dc-99cd-48a75002fd14 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:26.174134 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca99a58-1849-49dc-99cd-48a75002fd14 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:26.183842 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4c32b3-0407-43bb-8b4c-1b27a625dbae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:26.184963 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4c32b3-0407-43bb-8b4c-1b27a625dbae admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:26.185404 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4c32b3-0407-43bb-8b4c-1b27a625dbae admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:26.185997 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4c32b3-0407-43bb-8b4c-1b27a625dbae admin admin] PATH_INFO: `/v3/projects/ee9bc983690a4688903488d0968b324d/users/123f2d4be7324f4c9a336e8f6c3db951/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:26.223271 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4c32b3-0407-43bb-8b4c-1b27a625dbae admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee9bc983690a4688903488d0968b324d, user_id=123f2d4be7324f4c9a336e8f6c3db951, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:26.234839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4c32b3-0407-43bb-8b4c-1b27a625dbae admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:26.261089 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1171/2305] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:03:26 2022] PUT /identity/v3/projects/ee9bc983690a4688903488d0968b324d/users/123f2d4be7324f4c9a336e8f6c3db951/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:03:26.280561 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc96647-9f03-47d0-85af-13fae767daaa None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:26.280855 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc96647-9f03-47d0-85af-13fae767daaa None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:26.281130 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc96647-9f03-47d0-85af-13fae767daaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:26.313362 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-efc96647-9f03-47d0-85af-13fae767daaa None None] MFA Rules not processed for user `123f2d4be7324f4c9a336e8f6c3db951`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:26.314752 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-efc96647-9f03-47d0-85af-13fae767daaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:26.365250 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:26.399244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1135/2306] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:26 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:26.403527 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4c32b3-0407-43bb-8b4c-1b27a625dbae admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:26.452348 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:26.493881 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4c32b3-0407-43bb-8b4c-1b27a625dbae admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:26.501076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b97a8f7-19fe-4228-82a9-86c49a964355 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:26.501870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b97a8f7-19fe-4228-82a9-86c49a964355 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:26.502190 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b97a8f7-19fe-4228-82a9-86c49a964355 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:26.502715 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b97a8f7-19fe-4228-82a9-86c49a964355 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:26.503726 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b97a8f7-19fe-4228-82a9-86c49a964355 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:26.505528 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b97a8f7-19fe-4228-82a9-86c49a964355 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:26.549368 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1172/2307] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:03:26 2022] POST /identity/v3/projects => generated 377 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:26.553208 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc96647-9f03-47d0-85af-13fae767daaa None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:26.568504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc96647-9f03-47d0-85af-13fae767daaa None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:26.646954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b902f73-e739-4413-95f4-32038bcc3472 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:26.646954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b902f73-e739-4413-95f4-32038bcc3472 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:26.647766 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b902f73-e739-4413-95f4-32038bcc3472 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:26.648417 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b902f73-e739-4413-95f4-32038bcc3472 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:26.649407 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b902f73-e739-4413-95f4-32038bcc3472 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:26.651508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b902f73-e739-4413-95f4-32038bcc3472 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:26.684170 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1136/2308] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:03:26 2022] POST /identity/v3/users => generated 295 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:26.690004 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b97a8f7-19fe-4228-82a9-86c49a964355 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:26.709222 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b97a8f7-19fe-4228-82a9-86c49a964355 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:26.712531 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6d1b55-a6bb-4d14-94c1-7144c1755aa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:26.713249 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6d1b55-a6bb-4d14-94c1-7144c1755aa5 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:26.713528 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6d1b55-a6bb-4d14-94c1-7144c1755aa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:26.713801 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6d1b55-a6bb-4d14-94c1-7144c1755aa5 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:26.714449 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6d1b55-a6bb-4d14-94c1-7144c1755aa5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:26.716258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6d1b55-a6bb-4d14-94c1-7144c1755aa5 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:26.733242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1173/2309] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:26 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:26.737700 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b902f73-e739-4413-95f4-32038bcc3472 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:26.748914 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b902f73-e739-4413-95f4-32038bcc3472 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:26.755082 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc19ec42-2fb2-45ad-9f19-f2f9d1c19a94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:26.757166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc19ec42-2fb2-45ad-9f19-f2f9d1c19a94 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:26.757534 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc19ec42-2fb2-45ad-9f19-f2f9d1c19a94 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:26.757791 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc19ec42-2fb2-45ad-9f19-f2f9d1c19a94 admin admin] PATH_INFO: `/v3/projects/fee24a80cbcb429db63b5573f1f5fac1/users/2a657f8f5f8d445d955eb52012cdca6e/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:26.779880 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc19ec42-2fb2-45ad-9f19-f2f9d1c19a94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fee24a80cbcb429db63b5573f1f5fac1, user_id=2a657f8f5f8d445d955eb52012cdca6e, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:26.784379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc19ec42-2fb2-45ad-9f19-f2f9d1c19a94 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:26.814768 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1137/2310] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:03:26 2022] PUT /identity/v3/projects/fee24a80cbcb429db63b5573f1f5fac1/users/2a657f8f5f8d445d955eb52012cdca6e/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:26.819700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6d1b55-a6bb-4d14-94c1-7144c1755aa5 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:26.832310 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6d1b55-a6bb-4d14-94c1-7144c1755aa5 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:26.906124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa73629-3049-4659-95e8-e8a94454ed53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:26.906743 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa73629-3049-4659-95e8-e8a94454ed53 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:26.907018 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa73629-3049-4659-95e8-e8a94454ed53 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:26.907286 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa73629-3049-4659-95e8-e8a94454ed53 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:26.907963 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa73629-3049-4659-95e8-e8a94454ed53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:26.909784 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa73629-3049-4659-95e8-e8a94454ed53 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:26.928103 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1174/2311] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:26 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:26.930706 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc19ec42-2fb2-45ad-9f19-f2f9d1c19a94 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:26.941050 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc19ec42-2fb2-45ad-9f19-f2f9d1c19a94 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:26.946029 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1d0465-8282-4cc9-b74e-00be81102b7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:26.946682 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1d0465-8282-4cc9-b74e-00be81102b7c admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:26.946960 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1d0465-8282-4cc9-b74e-00be81102b7c admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:26.947229 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1d0465-8282-4cc9-b74e-00be81102b7c admin admin] PATH_INFO: `/v3/projects/fee24a80cbcb429db63b5573f1f5fac1/users/2a657f8f5f8d445d955eb52012cdca6e/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:26.949171 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1d0465-8282-4cc9-b74e-00be81102b7c admin admin] RBAC: Authorizing `identity:create_grant(project_id=fee24a80cbcb429db63b5573f1f5fac1, user_id=2a657f8f5f8d445d955eb52012cdca6e, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:26.951819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1d0465-8282-4cc9-b74e-00be81102b7c admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:26.962432 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1138/2312] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:03:26 2022] PUT /identity/v3/projects/fee24a80cbcb429db63b5573f1f5fac1/users/2a657f8f5f8d445d955eb52012cdca6e/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:26.973609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322b7fbf-5779-473d-aae0-56598a0ccdfb None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:26.973609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322b7fbf-5779-473d-aae0-56598a0ccdfb None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:26.974146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322b7fbf-5779-473d-aae0-56598a0ccdfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:27.005921 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-322b7fbf-5779-473d-aae0-56598a0ccdfb None None] MFA Rules not processed for user `2a657f8f5f8d445d955eb52012cdca6e`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:27.007919 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-322b7fbf-5779-473d-aae0-56598a0ccdfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:27.080602 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1175/2313] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:26 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:03:27.091661 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1d0465-8282-4cc9-b74e-00be81102b7c admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:27.093375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2c1d0465-8282-4cc9-b74e-00be81102b7c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:27.174417 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1d0465-8282-4cc9-b74e-00be81102b7c admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:27.178936 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab57fd18-3069-452c-bef3-a9cf735336e0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:27.179501 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab57fd18-3069-452c-bef3-a9cf735336e0 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:27.179832 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab57fd18-3069-452c-bef3-a9cf735336e0 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:27.180120 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab57fd18-3069-452c-bef3-a9cf735336e0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:27.237343 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:27.258176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab57fd18-3069-452c-bef3-a9cf735336e0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:27.260069 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab57fd18-3069-452c-bef3-a9cf735336e0 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:27.270883 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1139/2314] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:27 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 15:03:31.022775 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc4e54-eb48-4d67-b2a5-4d6156da8e96 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:31.023454 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc4e54-eb48-4d67-b2a5-4d6156da8e96 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:31.023454 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc4e54-eb48-4d67-b2a5-4d6156da8e96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:31.056542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-6bbc4e54-eb48-4d67-b2a5-4d6156da8e96 None None] MFA Rules not processed for user `c6dc53b3c9e041a7ae977b8f9c4fb478`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:31.058023 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-6bbc4e54-eb48-4d67-b2a5-4d6156da8e96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:31.086107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:31.136187 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1176/2315] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:31 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:03:31.140677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab57fd18-3069-452c-bef3-a9cf735336e0 service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:31.142186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ab57fd18-3069-452c-bef3-a9cf735336e0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:31.211120 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab57fd18-3069-452c-bef3-a9cf735336e0 service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:31.215871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec52770f-3383-4d99-8b4f-c3deb7ab9ce6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:31.216561 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec52770f-3383-4d99-8b4f-c3deb7ab9ce6 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:31.216842 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec52770f-3383-4d99-8b4f-c3deb7ab9ce6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:31.217113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec52770f-3383-4d99-8b4f-c3deb7ab9ce6 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:31.217756 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec52770f-3383-4d99-8b4f-c3deb7ab9ce6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:31.219769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec52770f-3383-4d99-8b4f-c3deb7ab9ce6 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:31.246638 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1140/2316] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:03:31 2022] POST /identity/v3/projects => generated 369 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:31.250743 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbc4e54-eb48-4d67-b2a5-4d6156da8e96 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:31.261883 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbc4e54-eb48-4d67-b2a5-4d6156da8e96 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:31.318664 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a77d42-8552-42c2-8d78-5087f6c8b22d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:31.320426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a77d42-8552-42c2-8d78-5087f6c8b22d admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:31.320657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a77d42-8552-42c2-8d78-5087f6c8b22d admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:31.320935 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a77d42-8552-42c2-8d78-5087f6c8b22d admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:31.321605 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a77d42-8552-42c2-8d78-5087f6c8b22d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:31.323515 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a77d42-8552-42c2-8d78-5087f6c8b22d admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:31.345135 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1177/2317] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:03:31 2022] POST /identity/v3/users => generated 291 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:31.349248 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec52770f-3383-4d99-8b4f-c3deb7ab9ce6 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:31.359494 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec52770f-3383-4d99-8b4f-c3deb7ab9ce6 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:31.364042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91c4ba5-a179-4eb4-9a83-5a01b0dd14dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:31.364627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91c4ba5-a179-4eb4-9a83-5a01b0dd14dd admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:31.364903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91c4ba5-a179-4eb4-9a83-5a01b0dd14dd admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:31.365174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91c4ba5-a179-4eb4-9a83-5a01b0dd14dd admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:31.365807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91c4ba5-a179-4eb4-9a83-5a01b0dd14dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:31.367381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91c4ba5-a179-4eb4-9a83-5a01b0dd14dd admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:31.377801 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1141/2318] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:31 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:31.382405 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a77d42-8552-42c2-8d78-5087f6c8b22d admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:31.392092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a77d42-8552-42c2-8d78-5087f6c8b22d admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:31.396731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfec4c2d-7f60-4b2f-9a3d-92b6bc653ad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:31.397934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfec4c2d-7f60-4b2f-9a3d-92b6bc653ad4 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:31.398205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfec4c2d-7f60-4b2f-9a3d-92b6bc653ad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:31.398475 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfec4c2d-7f60-4b2f-9a3d-92b6bc653ad4 admin admin] PATH_INFO: `/v3/projects/eb900e1a5dd74001934b483145f18364/users/7b9180028dd048179e4247fe2e64a83b/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:31.413377 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfec4c2d-7f60-4b2f-9a3d-92b6bc653ad4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb900e1a5dd74001934b483145f18364, user_id=7b9180028dd048179e4247fe2e64a83b, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:31.416057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfec4c2d-7f60-4b2f-9a3d-92b6bc653ad4 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:31.430192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1178/2319] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:03:31 2022] PUT /identity/v3/projects/eb900e1a5dd74001934b483145f18364/users/7b9180028dd048179e4247fe2e64a83b/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:31.434117 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91c4ba5-a179-4eb4-9a83-5a01b0dd14dd admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:31.445514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91c4ba5-a179-4eb4-9a83-5a01b0dd14dd admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:31.509760 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458b600c-1f11-4635-8c63-253d5e9b6682 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:31.510370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458b600c-1f11-4635-8c63-253d5e9b6682 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:31.510652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458b600c-1f11-4635-8c63-253d5e9b6682 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:31.511589 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458b600c-1f11-4635-8c63-253d5e9b6682 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:31.512244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458b600c-1f11-4635-8c63-253d5e9b6682 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:31.513846 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458b600c-1f11-4635-8c63-253d5e9b6682 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:31.524350 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1142/2320] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:31 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:31.528519 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfec4c2d-7f60-4b2f-9a3d-92b6bc653ad4 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:31.538934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfec4c2d-7f60-4b2f-9a3d-92b6bc653ad4 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:31.543830 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941c0fb5-f3f9-4ed8-9585-07fc4625bea2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:31.544565 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941c0fb5-f3f9-4ed8-9585-07fc4625bea2 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:31.544915 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941c0fb5-f3f9-4ed8-9585-07fc4625bea2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:31.545255 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941c0fb5-f3f9-4ed8-9585-07fc4625bea2 admin admin] PATH_INFO: `/v3/projects/eb900e1a5dd74001934b483145f18364/users/7b9180028dd048179e4247fe2e64a83b/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:31.547496 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941c0fb5-f3f9-4ed8-9585-07fc4625bea2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb900e1a5dd74001934b483145f18364, user_id=7b9180028dd048179e4247fe2e64a83b, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:31.550041 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941c0fb5-f3f9-4ed8-9585-07fc4625bea2 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:31.564695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1179/2321] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:03:31 2022] PUT /identity/v3/projects/eb900e1a5dd74001934b483145f18364/users/7b9180028dd048179e4247fe2e64a83b/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:32.348192 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458b600c-1f11-4635-8c63-253d5e9b6682 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:32.350577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-458b600c-1f11-4635-8c63-253d5e9b6682 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:32.426428 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458b600c-1f11-4635-8c63-253d5e9b6682 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:32.436949 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:32.490831 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea43d254-e5ee-4d66-bbfe-249c9beb3279 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:32.491507 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea43d254-e5ee-4d66-bbfe-249c9beb3279 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:32.491838 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea43d254-e5ee-4d66-bbfe-249c9beb3279 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:32.492162 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea43d254-e5ee-4d66-bbfe-249c9beb3279 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:32.574985 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea43d254-e5ee-4d66-bbfe-249c9beb3279 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:32.576892 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea43d254-e5ee-4d66-bbfe-249c9beb3279 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:32.590657 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1143/2322] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:32 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 243 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:03:36.345507 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6734fe7-bd34-449f-888a-8c6a27cab0ac None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:36.346028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6734fe7-bd34-449f-888a-8c6a27cab0ac None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:36.346028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6734fe7-bd34-449f-888a-8c6a27cab0ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:36.379750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-c6734fe7-bd34-449f-888a-8c6a27cab0ac None None] MFA Rules not processed for user `7b9180028dd048179e4247fe2e64a83b`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:36.381665 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-c6734fe7-bd34-449f-888a-8c6a27cab0ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:36.473142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:36.479705 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1180/2323] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:36 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:36.495932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea43d254-e5ee-4d66-bbfe-249c9beb3279 service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:36.497844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ea43d254-e5ee-4d66-bbfe-249c9beb3279 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:36.594711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea43d254-e5ee-4d66-bbfe-249c9beb3279 service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:36.601156 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe0d7ca-09b6-4c3a-8d33-619a446150c6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:36.601777 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe0d7ca-09b6-4c3a-8d33-619a446150c6 service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:36.602052 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe0d7ca-09b6-4c3a-8d33-619a446150c6 service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:36.602320 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe0d7ca-09b6-4c3a-8d33-619a446150c6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:36.666550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe0d7ca-09b6-4c3a-8d33-619a446150c6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:36.668202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe0d7ca-09b6-4c3a-8d33-619a446150c6 service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:36.673711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:36.680431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1144/2324] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:36 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:03:37.695990 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac4327d-de72-4fff-9b30-075ec67e368a None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:37.696476 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac4327d-de72-4fff-9b30-075ec67e368a None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:37.696690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac4327d-de72-4fff-9b30-075ec67e368a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:37.717191 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-5ac4327d-de72-4fff-9b30-075ec67e368a None None] MFA Rules not processed for user `c6dc53b3c9e041a7ae977b8f9c4fb478`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:37.718409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5ac4327d-de72-4fff-9b30-075ec67e368a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:37.725637 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1181/2325] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:37 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:40.079826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe0d7ca-09b6-4c3a-8d33-619a446150c6 service nova] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:40.081283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2fe0d7ca-09b6-4c3a-8d33-619a446150c6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:40.146763 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe0d7ca-09b6-4c3a-8d33-619a446150c6 service nova] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:40.151291 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685ae3ad-980e-4a65-be12-d387a7cb6e38 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:40.151932 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685ae3ad-980e-4a65-be12-d387a7cb6e38 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:40.152210 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685ae3ad-980e-4a65-be12-d387a7cb6e38 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:40.152479 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685ae3ad-980e-4a65-be12-d387a7cb6e38 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:40.154231 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-685ae3ad-980e-4a65-be12-d387a7cb6e38 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:40.223574 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685ae3ad-980e-4a65-be12-d387a7cb6e38 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:40.225798 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685ae3ad-980e-4a65-be12-d387a7cb6e38 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:40.236951 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1145/2326] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:03:43.666389 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac4327d-de72-4fff-9b30-075ec67e368a None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:43.666389 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5ac4327d-de72-4fff-9b30-075ec67e368a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:43.685285 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac4327d-de72-4fff-9b30-075ec67e368a None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:43.689898 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83a4cb5-b52d-4830-8421-5c39399afa1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:43.690528 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83a4cb5-b52d-4830-8421-5c39399afa1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:43.690810 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83a4cb5-b52d-4830-8421-5c39399afa1d admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:43.691092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83a4cb5-b52d-4830-8421-5c39399afa1d admin admin] PATH_INFO: `/v3/users/e1fb0b1657cd43baaa93a16ce6d9ec8f` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:43.692413 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83a4cb5-b52d-4830-8421-5c39399afa1d admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1fb0b1657cd43baaa93a16ce6d9ec8f)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:43.695217 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83a4cb5-b52d-4830-8421-5c39399afa1d admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:43.756116 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f83a4cb5-b52d-4830-8421-5c39399afa1d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1fb0b1657cd43baaa93a16ce6d9ec8f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:43.756858 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f83a4cb5-b52d-4830-8421-5c39399afa1d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1fb0b1657cd43baaa93a16ce6d9ec8f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:43.769035 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f83a4cb5-b52d-4830-8421-5c39399afa1d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1fb0b1657cd43baaa93a16ce6d9ec8f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:43.776031 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f83a4cb5-b52d-4830-8421-5c39399afa1d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1fb0b1657cd43baaa93a16ce6d9ec8f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:43.783298 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-f83a4cb5-b52d-4830-8421-5c39399afa1d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1fb0b1657cd43baaa93a16ce6d9ec8f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:43.789703 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1182/2327] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:03:43 2022] DELETE /identity/v3/users/e1fb0b1657cd43baaa93a16ce6d9ec8f => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:43.970025 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685ae3ad-980e-4a65-be12-d387a7cb6e38 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:43.989620 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685ae3ad-980e-4a65-be12-d387a7cb6e38 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:44.051476 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0001eae-d247-46be-b8e5-f2355cc04387 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:44.054188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0001eae-d247-46be-b8e5-f2355cc04387 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:44.054431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0001eae-d247-46be-b8e5-f2355cc04387 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:44.054711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0001eae-d247-46be-b8e5-f2355cc04387 admin admin] PATH_INFO: `/v3/projects/5119fe6e3c064da0a66218481a944aaa` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:44.056013 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0001eae-d247-46be-b8e5-f2355cc04387 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5119fe6e3c064da0a66218481a944aaa)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:44.058007 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0001eae-d247-46be-b8e5-f2355cc04387 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:44.104687 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d0001eae-d247-46be-b8e5-f2355cc04387 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5119fe6e3c064da0a66218481a944aaa', 'request_id': 'req-d0001eae-d247-46be-b8e5-f2355cc04387'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:44.107141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:44.112413 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d0001eae-d247-46be-b8e5-f2355cc04387 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5119fe6e3c064da0a66218481a944aaa', 'request_id': 'req-d0001eae-d247-46be-b8e5-f2355cc04387'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:44.120388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d0001eae-d247-46be-b8e5-f2355cc04387 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5119fe6e3c064da0a66218481a944aaa', 'request_id': 'req-d0001eae-d247-46be-b8e5-f2355cc04387'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:44.132059 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d0001eae-d247-46be-b8e5-f2355cc04387 admin admin] The token cache is being invalidate because project 5119fe6e3c064da0a66218481a944aaa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:03:44.132360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d0001eae-d247-46be-b8e5-f2355cc04387 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:44.133026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d0001eae-d247-46be-b8e5-f2355cc04387 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:44.134558 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1146/2328] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:03:43 2022] DELETE /identity/v3/projects/5119fe6e3c064da0a66218481a944aaa => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:44.140374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a7e620-fb25-4684-8c88-54113dbe5a56 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:44.140742 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a7e620-fb25-4684-8c88-54113dbe5a56 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:44.141139 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a7e620-fb25-4684-8c88-54113dbe5a56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:44.164105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-97a7e620-fb25-4684-8c88-54113dbe5a56 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:44.165267 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-97a7e620-fb25-4684-8c88-54113dbe5a56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:44.228978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1183/2329] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:44.236465 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d07707b-a131-4504-bc56-cb5de53331ef None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:44.236757 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d07707b-a131-4504-bc56-cb5de53331ef None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:44.237038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d07707b-a131-4504-bc56-cb5de53331ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:44.259776 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-7d07707b-a131-4504-bc56-cb5de53331ef None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:44.261032 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-7d07707b-a131-4504-bc56-cb5de53331ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:44.269391 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1147/2330] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:44.274829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a7e620-fb25-4684-8c88-54113dbe5a56 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:44.284743 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a7e620-fb25-4684-8c88-54113dbe5a56 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:44.290416 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a52556-5642-4a6c-9374-d833969388e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:44.291035 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a52556-5642-4a6c-9374-d833969388e8 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:44.291318 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a52556-5642-4a6c-9374-d833969388e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:44.291622 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a52556-5642-4a6c-9374-d833969388e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:44.292308 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a52556-5642-4a6c-9374-d833969388e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:03:44.292614 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a52556-5642-4a6c-9374-d833969388e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:44.294220 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a52556-5642-4a6c-9374-d833969388e8 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:44.306365 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1184/2331] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:03:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:03:44.310884 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d07707b-a131-4504-bc56-cb5de53331ef None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:44.321212 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d07707b-a131-4504-bc56-cb5de53331ef None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:44.325982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89997e12-11fb-46ba-b7d3-eba5c465d703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:44.326597 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89997e12-11fb-46ba-b7d3-eba5c465d703 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:44.326880 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89997e12-11fb-46ba-b7d3-eba5c465d703 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:44.327154 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89997e12-11fb-46ba-b7d3-eba5c465d703 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:44.327839 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89997e12-11fb-46ba-b7d3-eba5c465d703 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:44.329821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89997e12-11fb-46ba-b7d3-eba5c465d703 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:44.358165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1148/2332] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:03:44 2022] POST /identity/v3/projects => generated 383 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:44.362563 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a52556-5642-4a6c-9374-d833969388e8 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:44.372961 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a52556-5642-4a6c-9374-d833969388e8 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:44.423150 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:44.433983 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a789f8-6fbe-45ed-a4e8-96e09ea68d04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:44.435265 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a789f8-6fbe-45ed-a4e8-96e09ea68d04 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:44.435561 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a789f8-6fbe-45ed-a4e8-96e09ea68d04 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:44.435858 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a789f8-6fbe-45ed-a4e8-96e09ea68d04 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:44.436519 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a789f8-6fbe-45ed-a4e8-96e09ea68d04 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:44.438500 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a789f8-6fbe-45ed-a4e8-96e09ea68d04 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:44.462908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1185/2333] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:03:44 2022] POST /identity/v3/users => generated 292 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:44.464588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89997e12-11fb-46ba-b7d3-eba5c465d703 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:44.477569 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89997e12-11fb-46ba-b7d3-eba5c465d703 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:44.482417 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85de301b-ee3f-4c5b-b330-dd38e7b92649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:44.483077 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85de301b-ee3f-4c5b-b330-dd38e7b92649 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:44.483464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85de301b-ee3f-4c5b-b330-dd38e7b92649 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:44.483767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85de301b-ee3f-4c5b-b330-dd38e7b92649 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:44.484440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85de301b-ee3f-4c5b-b330-dd38e7b92649 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:44.486501 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85de301b-ee3f-4c5b-b330-dd38e7b92649 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:44.497209 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1149/2334] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:44 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:44.502093 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a789f8-6fbe-45ed-a4e8-96e09ea68d04 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:44.512720 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a789f8-6fbe-45ed-a4e8-96e09ea68d04 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:44.517545 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746ca055-9388-4751-aa45-3d9918a51272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:44.518177 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746ca055-9388-4751-aa45-3d9918a51272 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:44.518461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746ca055-9388-4751-aa45-3d9918a51272 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:44.518738 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746ca055-9388-4751-aa45-3d9918a51272 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:44.519384 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746ca055-9388-4751-aa45-3d9918a51272 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:44.521869 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746ca055-9388-4751-aa45-3d9918a51272 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:44.535420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1186/2335] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:44 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:44.540838 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85de301b-ee3f-4c5b-b330-dd38e7b92649 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:44.551672 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85de301b-ee3f-4c5b-b330-dd38e7b92649 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:44.560588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619730e1-d8f2-4ebf-ba06-7d62fc29ff9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:44.562223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619730e1-d8f2-4ebf-ba06-7d62fc29ff9c admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:44.563270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619730e1-d8f2-4ebf-ba06-7d62fc29ff9c admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:44.563270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619730e1-d8f2-4ebf-ba06-7d62fc29ff9c admin admin] PATH_INFO: `/v3/projects/484a2d46e66049edbb8efb13ae810e25/users/2fa86aaddd7d4f62a613c46c7ec296df/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:44.591856 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619730e1-d8f2-4ebf-ba06-7d62fc29ff9c admin admin] RBAC: Authorizing `identity:create_grant(project_id=484a2d46e66049edbb8efb13ae810e25, user_id=2fa86aaddd7d4f62a613c46c7ec296df, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:44.591856 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619730e1-d8f2-4ebf-ba06-7d62fc29ff9c admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:44.603670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1150/2336] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:03:44 2022] PUT /identity/v3/projects/484a2d46e66049edbb8efb13ae810e25/users/2fa86aaddd7d4f62a613c46c7ec296df/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:44.613759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18bf6a0-4f43-48df-bdd7-0438d8e5207b None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:44.614036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18bf6a0-4f43-48df-bdd7-0438d8e5207b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:44.614306 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18bf6a0-4f43-48df-bdd7-0438d8e5207b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:44.647021 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-d18bf6a0-4f43-48df-bdd7-0438d8e5207b None None] MFA Rules not processed for user `2fa86aaddd7d4f62a613c46c7ec296df`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:44.648338 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d18bf6a0-4f43-48df-bdd7-0438d8e5207b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:44.719778 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1187/2337] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:44 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:44.731257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619730e1-d8f2-4ebf-ba06-7d62fc29ff9c admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:44.733017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-619730e1-d8f2-4ebf-ba06-7d62fc29ff9c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:44.814605 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619730e1-d8f2-4ebf-ba06-7d62fc29ff9c admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:44.821912 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9468763f-1a27-4176-be2d-5b426cdd20f4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:44.822533 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9468763f-1a27-4176-be2d-5b426cdd20f4 service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:44.822825 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9468763f-1a27-4176-be2d-5b426cdd20f4 service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:44.823101 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9468763f-1a27-4176-be2d-5b426cdd20f4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:44.831371 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9468763f-1a27-4176-be2d-5b426cdd20f4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:44.833008 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9468763f-1a27-4176-be2d-5b426cdd20f4 service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:44.842743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1151/2338] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:44 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:03:45.256987 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18bf6a0-4f43-48df-bdd7-0438d8e5207b None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:45.259737 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d18bf6a0-4f43-48df-bdd7-0438d8e5207b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:45.331839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18bf6a0-4f43-48df-bdd7-0438d8e5207b None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:45.336771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4188e3c0-e982-4ffe-a117-bf7367111598 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:45.337473 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4188e3c0-e982-4ffe-a117-bf7367111598 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:45.337817 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4188e3c0-e982-4ffe-a117-bf7367111598 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:45.338156 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4188e3c0-e982-4ffe-a117-bf7367111598 admin admin] PATH_INFO: `/v3/users/f63fcc3f3e8246819c25f4c8a66e0d22` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:45.339339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4188e3c0-e982-4ffe-a117-bf7367111598 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f63fcc3f3e8246819c25f4c8a66e0d22)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:45.341488 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4188e3c0-e982-4ffe-a117-bf7367111598 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:45.391863 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-4188e3c0-e982-4ffe-a117-bf7367111598 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f63fcc3f3e8246819c25f4c8a66e0d22'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:45.392628 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-4188e3c0-e982-4ffe-a117-bf7367111598 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f63fcc3f3e8246819c25f4c8a66e0d22'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:45.403561 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=88,keystone:INSERT=3,keystone:DELETE=9 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:45.405075 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-4188e3c0-e982-4ffe-a117-bf7367111598 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f63fcc3f3e8246819c25f4c8a66e0d22'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:45.415464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-4188e3c0-e982-4ffe-a117-bf7367111598 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f63fcc3f3e8246819c25f4c8a66e0d22'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:45.422644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-4188e3c0-e982-4ffe-a117-bf7367111598 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f63fcc3f3e8246819c25f4c8a66e0d22'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:45.429595 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1188/2339] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:03:45 2022] DELETE /identity/v3/users/f63fcc3f3e8246819c25f4c8a66e0d22 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:45.435088 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5817760e-57c8-4bef-acbd-44acb855389b None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:45.435088 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5817760e-57c8-4bef-acbd-44acb855389b None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:45.435088 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5817760e-57c8-4bef-acbd-44acb855389b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:45.454791 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-5817760e-57c8-4bef-acbd-44acb855389b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:45.455984 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5817760e-57c8-4bef-acbd-44acb855389b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:45.517383 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1152/2340] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:45.527941 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4188e3c0-e982-4ffe-a117-bf7367111598 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:45.529641 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-4188e3c0-e982-4ffe-a117-bf7367111598 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:45.592852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4188e3c0-e982-4ffe-a117-bf7367111598 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:45.596708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2563cd6d-e9eb-4c4b-9a64-fb71f9c5b2a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:45.597324 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2563cd6d-e9eb-4c4b-9a64-fb71f9c5b2a4 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:45.597604 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2563cd6d-e9eb-4c4b-9a64-fb71f9c5b2a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:45.597879 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2563cd6d-e9eb-4c4b-9a64-fb71f9c5b2a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:45.609786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2563cd6d-e9eb-4c4b-9a64-fb71f9c5b2a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:45.611200 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2563cd6d-e9eb-4c4b-9a64-fb71f9c5b2a4 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:45.622973 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1189/2341] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:03:45.897592 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5817760e-57c8-4bef-acbd-44acb855389b None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:45.899071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5817760e-57c8-4bef-acbd-44acb855389b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:45.906562 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:DELETE=3,keystone:SELECT=95,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:45.915502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5817760e-57c8-4bef-acbd-44acb855389b None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:45.921964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa793a1e-eb42-47fb-8680-93bba3ed8957 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:45.922638 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa793a1e-eb42-47fb-8680-93bba3ed8957 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:45.922925 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa793a1e-eb42-47fb-8680-93bba3ed8957 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:45.923203 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa793a1e-eb42-47fb-8680-93bba3ed8957 admin admin] PATH_INFO: `/v3/projects/e774aa6b8fe944c5a408a2910caafd7f` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:45.924444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa793a1e-eb42-47fb-8680-93bba3ed8957 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e774aa6b8fe944c5a408a2910caafd7f)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:45.926371 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa793a1e-eb42-47fb-8680-93bba3ed8957 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:45.979488 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-fa793a1e-eb42-47fb-8680-93bba3ed8957 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e774aa6b8fe944c5a408a2910caafd7f', 'request_id': 'req-fa793a1e-eb42-47fb-8680-93bba3ed8957'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:45.989686 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-fa793a1e-eb42-47fb-8680-93bba3ed8957 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e774aa6b8fe944c5a408a2910caafd7f', 'request_id': 'req-fa793a1e-eb42-47fb-8680-93bba3ed8957'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:45.994949 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-fa793a1e-eb42-47fb-8680-93bba3ed8957 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e774aa6b8fe944c5a408a2910caafd7f', 'request_id': 'req-fa793a1e-eb42-47fb-8680-93bba3ed8957'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:46.006270 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-fa793a1e-eb42-47fb-8680-93bba3ed8957 admin admin] The token cache is being invalidate because project e774aa6b8fe944c5a408a2910caafd7f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:03:46.006561 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-fa793a1e-eb42-47fb-8680-93bba3ed8957 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:46.007222 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-fa793a1e-eb42-47fb-8680-93bba3ed8957 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:46.008735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1153/2342] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:03:45 2022] DELETE /identity/v3/projects/e774aa6b8fe944c5a408a2910caafd7f => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:46.014673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1530a1-572d-4fd3-a77b-d523b099d831 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:46.015182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1530a1-572d-4fd3-a77b-d523b099d831 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:46.015295 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1530a1-572d-4fd3-a77b-d523b099d831 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:46.035296 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-2f1530a1-572d-4fd3-a77b-d523b099d831 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:46.037163 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2f1530a1-572d-4fd3-a77b-d523b099d831 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:46.100624 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1190/2343] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:46.108797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823d4a85-6f1a-451a-90a0-215e69849d9f None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:46.109078 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823d4a85-6f1a-451a-90a0-215e69849d9f None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:46.109350 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823d4a85-6f1a-451a-90a0-215e69849d9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:46.128914 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-823d4a85-6f1a-451a-90a0-215e69849d9f None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:46.130121 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-823d4a85-6f1a-451a-90a0-215e69849d9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:46.138071 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1154/2344] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:46.142639 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1530a1-572d-4fd3-a77b-d523b099d831 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:46.153166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1530a1-572d-4fd3-a77b-d523b099d831 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:46.158506 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac1e010-1b0c-43af-a63a-2d7431d8c5dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:46.159119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac1e010-1b0c-43af-a63a-2d7431d8c5dc admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:46.159399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac1e010-1b0c-43af-a63a-2d7431d8c5dc admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:46.159711 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac1e010-1b0c-43af-a63a-2d7431d8c5dc admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:46.160364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac1e010-1b0c-43af-a63a-2d7431d8c5dc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:03:46.160668 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac1e010-1b0c-43af-a63a-2d7431d8c5dc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:46.162960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac1e010-1b0c-43af-a63a-2d7431d8c5dc admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:46.174975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1191/2345] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:03:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:03:46.179266 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823d4a85-6f1a-451a-90a0-215e69849d9f None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:46.188779 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823d4a85-6f1a-451a-90a0-215e69849d9f None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:46.200048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a616a253-4356-48c4-98f5-38e54035719a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:46.200669 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a616a253-4356-48c4-98f5-38e54035719a admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:46.200943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a616a253-4356-48c4-98f5-38e54035719a admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:46.201208 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a616a253-4356-48c4-98f5-38e54035719a admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:46.201847 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a616a253-4356-48c4-98f5-38e54035719a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:46.203953 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a616a253-4356-48c4-98f5-38e54035719a admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:46.232502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1155/2346] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:03:46 2022] POST /identity/v3/projects => generated 377 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:46.237041 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac1e010-1b0c-43af-a63a-2d7431d8c5dc admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:46.248715 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac1e010-1b0c-43af-a63a-2d7431d8c5dc admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:46.316792 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476e6085-1102-45b4-8e09-d492f669fdec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:46.317479 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476e6085-1102-45b4-8e09-d492f669fdec admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:46.318387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476e6085-1102-45b4-8e09-d492f669fdec admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:46.318801 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476e6085-1102-45b4-8e09-d492f669fdec admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:46.319786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-476e6085-1102-45b4-8e09-d492f669fdec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:46.321839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-476e6085-1102-45b4-8e09-d492f669fdec admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:46.340330 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:46.348735 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1192/2347] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:03:46 2022] POST /identity/v3/users => generated 289 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:46.353253 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a616a253-4356-48c4-98f5-38e54035719a admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:46.366788 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a616a253-4356-48c4-98f5-38e54035719a admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:46.371754 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0259b80a-3350-47e4-b9d2-c62ef84b4bde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:46.372380 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0259b80a-3350-47e4-b9d2-c62ef84b4bde admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:46.372655 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0259b80a-3350-47e4-b9d2-c62ef84b4bde admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:46.372926 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0259b80a-3350-47e4-b9d2-c62ef84b4bde admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:46.373592 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0259b80a-3350-47e4-b9d2-c62ef84b4bde admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:46.376443 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0259b80a-3350-47e4-b9d2-c62ef84b4bde admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:46.388305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1156/2348] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:46 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:03:46.393551 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476e6085-1102-45b4-8e09-d492f669fdec admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:46.406136 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476e6085-1102-45b4-8e09-d492f669fdec admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:46.411996 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25120e0-3134-4dba-a228-424549a6b4bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:46.412876 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25120e0-3134-4dba-a228-424549a6b4bc admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:46.413164 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25120e0-3134-4dba-a228-424549a6b4bc admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:46.413486 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25120e0-3134-4dba-a228-424549a6b4bc admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:46.414089 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25120e0-3134-4dba-a228-424549a6b4bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:46.415754 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25120e0-3134-4dba-a228-424549a6b4bc admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:46.427617 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1193/2349] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:46 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:03:46.431871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0259b80a-3350-47e4-b9d2-c62ef84b4bde admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:46.445362 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0259b80a-3350-47e4-b9d2-c62ef84b4bde admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:46.452485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c72e653-091c-40ed-bf11-a4eccf31a2be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:46.453866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c72e653-091c-40ed-bf11-a4eccf31a2be admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:46.453866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c72e653-091c-40ed-bf11-a4eccf31a2be admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:46.453866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c72e653-091c-40ed-bf11-a4eccf31a2be admin admin] PATH_INFO: `/v3/projects/d7f5a9331dd54d9e8b5033234cb1077a/users/f0b358738b5a46d8bc1db1da68040c8f/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:46.470732 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c72e653-091c-40ed-bf11-a4eccf31a2be admin admin] RBAC: Authorizing `identity:create_grant(project_id=d7f5a9331dd54d9e8b5033234cb1077a, user_id=f0b358738b5a46d8bc1db1da68040c8f, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:46.473445 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c72e653-091c-40ed-bf11-a4eccf31a2be admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:46.489376 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1157/2350] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:03:46 2022] PUT /identity/v3/projects/d7f5a9331dd54d9e8b5033234cb1077a/users/f0b358738b5a46d8bc1db1da68040c8f/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:03:46.500487 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eb872f-1f74-4ec7-93ec-257df67d5f58 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:46.500852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eb872f-1f74-4ec7-93ec-257df67d5f58 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:46.501387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eb872f-1f74-4ec7-93ec-257df67d5f58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:46.542011 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-d4eb872f-1f74-4ec7-93ec-257df67d5f58 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:46.544022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d4eb872f-1f74-4ec7-93ec-257df67d5f58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:46.630989 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1194/2351] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:46.641559 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c72e653-091c-40ed-bf11-a4eccf31a2be admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:46.643648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-7c72e653-091c-40ed-bf11-a4eccf31a2be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:46.755978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c72e653-091c-40ed-bf11-a4eccf31a2be admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:46.759672 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ec3662-a116-4e3f-8a62-504536d69adf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:46.760252 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ec3662-a116-4e3f-8a62-504536d69adf service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:46.760534 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ec3662-a116-4e3f-8a62-504536d69adf service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:46.760809 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ec3662-a116-4e3f-8a62-504536d69adf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:46.769955 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ec3662-a116-4e3f-8a62-504536d69adf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:46.772243 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ec3662-a116-4e3f-8a62-504536d69adf service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:46.781562 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1158/2352] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:03:50.744654 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c338c46-01cc-46e6-a395-9ba22d0f7c74 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:50.745272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c338c46-01cc-46e6-a395-9ba22d0f7c74 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:50.746091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c338c46-01cc-46e6-a395-9ba22d0f7c74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:50.780872 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-9c338c46-01cc-46e6-a395-9ba22d0f7c74 None None] MFA Rules not processed for user `f0b358738b5a46d8bc1db1da68040c8f`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:50.782096 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-9c338c46-01cc-46e6-a395-9ba22d0f7c74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:50.872822 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1195/2353] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:50 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:50.877523 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ec3662-a116-4e3f-8a62-504536d69adf service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:50.888241 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ec3662-a116-4e3f-8a62-504536d69adf service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:50.895669 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9104af83-9bb1-4b9b-85c8-368232f4e6d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:50.897121 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9104af83-9bb1-4b9b-85c8-368232f4e6d9 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:50.897481 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9104af83-9bb1-4b9b-85c8-368232f4e6d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:50.897873 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9104af83-9bb1-4b9b-85c8-368232f4e6d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:50.898627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9104af83-9bb1-4b9b-85c8-368232f4e6d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:50.900810 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9104af83-9bb1-4b9b-85c8-368232f4e6d9 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:50.943668 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1159/2354] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:03:50 2022] POST /identity/v3/projects => generated 377 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:50.948108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c338c46-01cc-46e6-a395-9ba22d0f7c74 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:50.963182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c338c46-01cc-46e6-a395-9ba22d0f7c74 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:51.040817 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483e4f3e-f118-4877-9243-21db844f9a0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:51.041436 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483e4f3e-f118-4877-9243-21db844f9a0a admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:51.041714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483e4f3e-f118-4877-9243-21db844f9a0a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:51.041984 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483e4f3e-f118-4877-9243-21db844f9a0a admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:51.042629 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483e4f3e-f118-4877-9243-21db844f9a0a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:51.044594 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483e4f3e-f118-4877-9243-21db844f9a0a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:51.072953 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1196/2355] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:03:50 2022] POST /identity/v3/users => generated 295 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:03:51.076304 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9104af83-9bb1-4b9b-85c8-368232f4e6d9 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:51.082000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:51.087796 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9104af83-9bb1-4b9b-85c8-368232f4e6d9 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:51.094068 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b6a2b6-0e4c-4983-9f3a-a3c54d7be3c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:51.094880 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b6a2b6-0e4c-4983-9f3a-a3c54d7be3c2 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:51.095222 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b6a2b6-0e4c-4983-9f3a-a3c54d7be3c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:51.095601 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b6a2b6-0e4c-4983-9f3a-a3c54d7be3c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:51.096319 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b6a2b6-0e4c-4983-9f3a-a3c54d7be3c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:51.097994 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b6a2b6-0e4c-4983-9f3a-a3c54d7be3c2 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:51.109945 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1160/2356] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:51 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:51.114744 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483e4f3e-f118-4877-9243-21db844f9a0a admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:51.125632 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483e4f3e-f118-4877-9243-21db844f9a0a admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:51.131663 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73594e4-47ec-406c-a97b-4185bf49c002 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:51.132335 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73594e4-47ec-406c-a97b-4185bf49c002 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:51.132635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73594e4-47ec-406c-a97b-4185bf49c002 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:51.132917 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73594e4-47ec-406c-a97b-4185bf49c002 admin admin] PATH_INFO: `/v3/projects/f1c52c7d41be4e1aa934c498760d7403/users/5324a9ce10ae4460ae36bac43f35c1c6/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:51.136350 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:51.150264 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73594e4-47ec-406c-a97b-4185bf49c002 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1c52c7d41be4e1aa934c498760d7403, user_id=5324a9ce10ae4460ae36bac43f35c1c6, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:51.153035 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73594e4-47ec-406c-a97b-4185bf49c002 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:51.166279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1197/2357] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:03:51 2022] PUT /identity/v3/projects/f1c52c7d41be4e1aa934c498760d7403/users/5324a9ce10ae4460ae36bac43f35c1c6/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:51.170396 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b6a2b6-0e4c-4983-9f3a-a3c54d7be3c2 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:51.181278 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b6a2b6-0e4c-4983-9f3a-a3c54d7be3c2 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:51.247739 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad16d1a6-c9d9-4f5f-9090-333312de5694 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:51.248600 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad16d1a6-c9d9-4f5f-9090-333312de5694 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:51.249006 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad16d1a6-c9d9-4f5f-9090-333312de5694 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:51.249447 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad16d1a6-c9d9-4f5f-9090-333312de5694 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:51.250290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad16d1a6-c9d9-4f5f-9090-333312de5694 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:51.251946 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad16d1a6-c9d9-4f5f-9090-333312de5694 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:51.262587 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1161/2358] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:51 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:03:51.266693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73594e4-47ec-406c-a97b-4185bf49c002 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:51.277006 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73594e4-47ec-406c-a97b-4185bf49c002 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:51.281829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59a313c-da70-41da-a636-aca13e587f99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:51.282479 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59a313c-da70-41da-a636-aca13e587f99 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:51.282757 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59a313c-da70-41da-a636-aca13e587f99 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:51.283026 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59a313c-da70-41da-a636-aca13e587f99 admin admin] PATH_INFO: `/v3/projects/f1c52c7d41be4e1aa934c498760d7403/users/5324a9ce10ae4460ae36bac43f35c1c6/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:51.284994 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59a313c-da70-41da-a636-aca13e587f99 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1c52c7d41be4e1aa934c498760d7403, user_id=5324a9ce10ae4460ae36bac43f35c1c6, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:51.287785 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59a313c-da70-41da-a636-aca13e587f99 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:51.297868 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1198/2359] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:03:51 2022] PUT /identity/v3/projects/f1c52c7d41be4e1aa934c498760d7403/users/5324a9ce10ae4460ae36bac43f35c1c6/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:55.820247 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad16d1a6-c9d9-4f5f-9090-333312de5694 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:55.820247 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ad16d1a6-c9d9-4f5f-9090-333312de5694 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:55.859341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad16d1a6-c9d9-4f5f-9090-333312de5694 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:56.011786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbf9989-606c-4e66-8620-9753a26e78a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:56.012436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbf9989-606c-4e66-8620-9753a26e78a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:56.012757 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbf9989-606c-4e66-8620-9753a26e78a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:56.013039 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbf9989-606c-4e66-8620-9753a26e78a6 admin admin] PATH_INFO: `/v3/users/2fa86aaddd7d4f62a613c46c7ec296df` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:56.014484 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bbf9989-606c-4e66-8620-9753a26e78a6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2fa86aaddd7d4f62a613c46c7ec296df)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:56.016542 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bbf9989-606c-4e66-8620-9753a26e78a6 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:56.105739 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8bbf9989-606c-4e66-8620-9753a26e78a6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2fa86aaddd7d4f62a613c46c7ec296df'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:56.114616 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8bbf9989-606c-4e66-8620-9753a26e78a6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2fa86aaddd7d4f62a613c46c7ec296df'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:56.115654 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8bbf9989-606c-4e66-8620-9753a26e78a6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2fa86aaddd7d4f62a613c46c7ec296df'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:56.116890 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8bbf9989-606c-4e66-8620-9753a26e78a6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2fa86aaddd7d4f62a613c46c7ec296df'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:56.131473 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-8bbf9989-606c-4e66-8620-9753a26e78a6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2fa86aaddd7d4f62a613c46c7ec296df'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:56.151442 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1162/2360] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:03:55 2022] DELETE /identity/v3/users/2fa86aaddd7d4f62a613c46c7ec296df => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:56.156259 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adcc9c1-f8e3-458f-87a2-ce40f96631fb None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:56.157350 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adcc9c1-f8e3-458f-87a2-ce40f96631fb None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:56.157658 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adcc9c1-f8e3-458f-87a2-ce40f96631fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:56.187585 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-5adcc9c1-f8e3-458f-87a2-ce40f96631fb None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:56.189090 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5adcc9c1-f8e3-458f-87a2-ce40f96631fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:56.280045 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1199/2361] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:56.280877 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbf9989-606c-4e66-8620-9753a26e78a6 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:56.282476 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-8bbf9989-606c-4e66-8620-9753a26e78a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:56.298806 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:56.376100 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbf9989-606c-4e66-8620-9753a26e78a6 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:56.381872 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e869dabc-8104-46b8-8848-578df8f532dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:56.382702 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e869dabc-8104-46b8-8848-578df8f532dc service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:56.383140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e869dabc-8104-46b8-8848-578df8f532dc service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:56.383613 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e869dabc-8104-46b8-8848-578df8f532dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:56.396191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e869dabc-8104-46b8-8848-578df8f532dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:56.398019 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e869dabc-8104-46b8-8848-578df8f532dc service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:56.409496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1163/2362] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 15:03:56.846353 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c61b165-72fd-4691-a4db-6d3056376c11 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:56.846792 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c61b165-72fd-4691-a4db-6d3056376c11 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:56.847211 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c61b165-72fd-4691-a4db-6d3056376c11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:56.874132 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e869dabc-8104-46b8-8848-578df8f532dc service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:56.875757 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e869dabc-8104-46b8-8848-578df8f532dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:56.879147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-3c61b165-72fd-4691-a4db-6d3056376c11 None None] MFA Rules not processed for user `5324a9ce10ae4460ae36bac43f35c1c6`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:56.881681 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-3c61b165-72fd-4691-a4db-6d3056376c11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:56.890081 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e869dabc-8104-46b8-8848-578df8f532dc service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:56.896003 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2c3d84-0400-4484-ae42-db37992098cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:56.896650 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2c3d84-0400-4484-ae42-db37992098cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:56.896929 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2c3d84-0400-4484-ae42-db37992098cc admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:56.897203 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2c3d84-0400-4484-ae42-db37992098cc admin admin] PATH_INFO: `/v3/projects/484a2d46e66049edbb8efb13ae810e25` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:56.898437 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2c3d84-0400-4484-ae42-db37992098cc admin admin] RBAC: Authorizing `identity:delete_project(project_id=484a2d46e66049edbb8efb13ae810e25)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:56.900602 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2c3d84-0400-4484-ae42-db37992098cc admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:56.953670 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2d2c3d84-0400-4484-ae42-db37992098cc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '484a2d46e66049edbb8efb13ae810e25', 'request_id': 'req-2d2c3d84-0400-4484-ae42-db37992098cc'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:56.964914 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2d2c3d84-0400-4484-ae42-db37992098cc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '484a2d46e66049edbb8efb13ae810e25', 'request_id': 'req-2d2c3d84-0400-4484-ae42-db37992098cc'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:56.975608 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2d2c3d84-0400-4484-ae42-db37992098cc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '484a2d46e66049edbb8efb13ae810e25', 'request_id': 'req-2d2c3d84-0400-4484-ae42-db37992098cc'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:56.975969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1200/2363] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:56 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:56.990558 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c61b165-72fd-4691-a4db-6d3056376c11 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:56.992014 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-3c61b165-72fd-4691-a4db-6d3056376c11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:56.992821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2d2c3d84-0400-4484-ae42-db37992098cc admin admin] The token cache is being invalidate because project 484a2d46e66049edbb8efb13ae810e25 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:03:56.993636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2d2c3d84-0400-4484-ae42-db37992098cc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:56.994300 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2d2c3d84-0400-4484-ae42-db37992098cc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:56.995872 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1164/2364] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:03:56 2022] DELETE /identity/v3/projects/484a2d46e66049edbb8efb13ae810e25 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:03:57.000471 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e5e06d-102e-465a-a828-f7f1a06c0174 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:57.000760 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e5e06d-102e-465a-a828-f7f1a06c0174 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:57.001043 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e5e06d-102e-465a-a828-f7f1a06c0174 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:57.022109 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-85e5e06d-102e-465a-a828-f7f1a06c0174 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:57.023973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-85e5e06d-102e-465a-a828-f7f1a06c0174 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:57.053939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:57.079350 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c61b165-72fd-4691-a4db-6d3056376c11 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:57.089738 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1165/2365] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:03:57.099002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6bd9e2-1406-4b78-9f67-07c67eaf0356 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:57.099293 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6bd9e2-1406-4b78-9f67-07c67eaf0356 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:57.099592 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6bd9e2-1406-4b78-9f67-07c67eaf0356 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:57.119970 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-fb6bd9e2-1406-4b78-9f67-07c67eaf0356 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:57.121188 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-fb6bd9e2-1406-4b78-9f67-07c67eaf0356 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:57.129219 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1166/2366] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:03:57.134150 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6bd9e2-1406-4b78-9f67-07c67eaf0356 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:57.143232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6bd9e2-1406-4b78-9f67-07c67eaf0356 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:57.147980 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4a5c51-bb66-4812-abe1-19f9ceadabf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:57.148477 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff68736-c780-4199-a9fa-f55fb44b8165 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:57.148611 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4a5c51-bb66-4812-abe1-19f9ceadabf8 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:57.148834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4a5c51-bb66-4812-abe1-19f9ceadabf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:57.149109 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4a5c51-bb66-4812-abe1-19f9ceadabf8 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:57.149205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff68736-c780-4199-a9fa-f55fb44b8165 service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:57.149387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff68736-c780-4199-a9fa-f55fb44b8165 service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:57.149661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff68736-c780-4199-a9fa-f55fb44b8165 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:57.149802 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4a5c51-bb66-4812-abe1-19f9ceadabf8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:03:57.150044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4a5c51-bb66-4812-abe1-19f9ceadabf8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:57.151398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-4ff68736-c780-4199-a9fa-f55fb44b8165 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:57.151697 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4a5c51-bb66-4812-abe1-19f9ceadabf8 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:57.163678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1167/2367] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:03:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 15:03:57.167980 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4a5c51-bb66-4812-abe1-19f9ceadabf8 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:57.178996 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4a5c51-bb66-4812-abe1-19f9ceadabf8 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:57.183848 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f2800-c198-42b0-9a3d-901dc91728e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:57.184435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f2800-c198-42b0-9a3d-901dc91728e7 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:57.184721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f2800-c198-42b0-9a3d-901dc91728e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:57.184994 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f2800-c198-42b0-9a3d-901dc91728e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:57.185617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659f2800-c198-42b0-9a3d-901dc91728e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:57.187514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659f2800-c198-42b0-9a3d-901dc91728e7 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:57.200795 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:57.211695 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff68736-c780-4199-a9fa-f55fb44b8165 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:57.213289 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff68736-c780-4199-a9fa-f55fb44b8165 service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:57.218566 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1168/2368] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:03:57 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:57.222563 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1201/2369] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:56 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:03:57.222921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f2800-c198-42b0-9a3d-901dc91728e7 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:57.287991 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f2800-c198-42b0-9a3d-901dc91728e7 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:57.292811 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed5d6f7-3930-4e05-a72f-1e8076da0e62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:57.293420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed5d6f7-3930-4e05-a72f-1e8076da0e62 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:57.293703 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed5d6f7-3930-4e05-a72f-1e8076da0e62 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:57.293979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed5d6f7-3930-4e05-a72f-1e8076da0e62 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:57.294617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed5d6f7-3930-4e05-a72f-1e8076da0e62 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:57.296588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed5d6f7-3930-4e05-a72f-1e8076da0e62 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:57.324855 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1169/2370] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:03:57 2022] POST /identity/v3/users => generated 293 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:57.326097 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff68736-c780-4199-a9fa-f55fb44b8165 service nova] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:57.337000 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff68736-c780-4199-a9fa-f55fb44b8165 service nova] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:57.342339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1129c180-a1f7-4841-95c4-8908c4edbf48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:57.343011 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1129c180-a1f7-4841-95c4-8908c4edbf48 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:57.343406 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1129c180-a1f7-4841-95c4-8908c4edbf48 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:57.343822 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1129c180-a1f7-4841-95c4-8908c4edbf48 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:57.344543 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1129c180-a1f7-4841-95c4-8908c4edbf48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:57.346284 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1129c180-a1f7-4841-95c4-8908c4edbf48 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:57.357134 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1202/2371] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:57.361161 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed5d6f7-3930-4e05-a72f-1e8076da0e62 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:57.372619 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed5d6f7-3930-4e05-a72f-1e8076da0e62 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:57.377292 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793f3142-60b0-4d54-bf85-dc36cfead6d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:57.377896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793f3142-60b0-4d54-bf85-dc36cfead6d7 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:57.378174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793f3142-60b0-4d54-bf85-dc36cfead6d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:57.378458 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793f3142-60b0-4d54-bf85-dc36cfead6d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:57.379077 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793f3142-60b0-4d54-bf85-dc36cfead6d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:57.381272 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793f3142-60b0-4d54-bf85-dc36cfead6d7 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:57.392392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1170/2372] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:57 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:03:57.396892 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1129c180-a1f7-4841-95c4-8908c4edbf48 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:57.406864 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1129c180-a1f7-4841-95c4-8908c4edbf48 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:57.413505 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2c5e10-2bbb-4bb3-9aa5-1d2efcb15950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:57.413700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2c5e10-2bbb-4bb3-9aa5-1d2efcb15950 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:57.414262 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2c5e10-2bbb-4bb3-9aa5-1d2efcb15950 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:57.414262 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2c5e10-2bbb-4bb3-9aa5-1d2efcb15950 admin admin] PATH_INFO: `/v3/projects/001f1cca96b243ab9fb2fe8c555ada77/users/ec8a3ee7912c4bd1bea33d1628d59981/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:57.435203 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2c5e10-2bbb-4bb3-9aa5-1d2efcb15950 admin admin] RBAC: Authorizing `identity:create_grant(project_id=001f1cca96b243ab9fb2fe8c555ada77, user_id=ec8a3ee7912c4bd1bea33d1628d59981, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:57.438093 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2c5e10-2bbb-4bb3-9aa5-1d2efcb15950 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:57.454896 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1203/2373] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:03:57 2022] PUT /identity/v3/projects/001f1cca96b243ab9fb2fe8c555ada77/users/ec8a3ee7912c4bd1bea33d1628d59981/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:03:57.466149 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0409cd-6918-4c4e-a149-4d3dc07d27ca None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:57.466430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0409cd-6918-4c4e-a149-4d3dc07d27ca None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:57.466702 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0409cd-6918-4c4e-a149-4d3dc07d27ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:57.503646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-5e0409cd-6918-4c4e-a149-4d3dc07d27ca None None] MFA Rules not processed for user `ec8a3ee7912c4bd1bea33d1628d59981`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:57.505184 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5e0409cd-6918-4c4e-a149-4d3dc07d27ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:57.616613 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1171/2374] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:57 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:57.617939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2c5e10-2bbb-4bb3-9aa5-1d2efcb15950 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:57.709774 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:57.716148 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2c5e10-2bbb-4bb3-9aa5-1d2efcb15950 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:57.722421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beba4984-c2f9-42bb-b618-5ddff51c975b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:57.723829 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beba4984-c2f9-42bb-b618-5ddff51c975b admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:57.724234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beba4984-c2f9-42bb-b618-5ddff51c975b admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:57.725274 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beba4984-c2f9-42bb-b618-5ddff51c975b admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:57.725962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beba4984-c2f9-42bb-b618-5ddff51c975b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:57.729018 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beba4984-c2f9-42bb-b618-5ddff51c975b admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:57.770835 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1204/2375] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:03:57 2022] POST /identity/v3/projects => generated 387 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:03:57.775267 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0409cd-6918-4c4e-a149-4d3dc07d27ca None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:57.790967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0409cd-6918-4c4e-a149-4d3dc07d27ca None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:57.859635 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afae3790-9e38-41cd-a6ee-9a235d10cacf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:57.860392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afae3790-9e38-41cd-a6ee-9a235d10cacf admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:57.860678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afae3790-9e38-41cd-a6ee-9a235d10cacf admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:57.861118 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afae3790-9e38-41cd-a6ee-9a235d10cacf admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:57.861587 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afae3790-9e38-41cd-a6ee-9a235d10cacf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:57.863736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afae3790-9e38-41cd-a6ee-9a235d10cacf admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:57.872628 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:57.892542 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1172/2376] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:03:57 2022] POST /identity/v3/users => generated 300 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:03:57.897197 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beba4984-c2f9-42bb-b618-5ddff51c975b admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:57.912034 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beba4984-c2f9-42bb-b618-5ddff51c975b admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:57.918743 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9578f32-5ca1-4dd4-96f0-a017d09e9967 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:57.919430 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9578f32-5ca1-4dd4-96f0-a017d09e9967 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:57.919782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9578f32-5ca1-4dd4-96f0-a017d09e9967 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:57.920058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9578f32-5ca1-4dd4-96f0-a017d09e9967 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:57.921285 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9578f32-5ca1-4dd4-96f0-a017d09e9967 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:57.922938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9578f32-5ca1-4dd4-96f0-a017d09e9967 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:57.948684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1205/2377] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:57 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:03:57.952850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afae3790-9e38-41cd-a6ee-9a235d10cacf admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:57.966757 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afae3790-9e38-41cd-a6ee-9a235d10cacf admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:57.972418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e8d414-b294-4f4f-8a73-e754914b43c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:57.973114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e8d414-b294-4f4f-8a73-e754914b43c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:57.973934 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e8d414-b294-4f4f-8a73-e754914b43c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:57.974212 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e8d414-b294-4f4f-8a73-e754914b43c2 admin admin] PATH_INFO: `/v3/projects/744ac10864bf4318bc8ee17d436a0f5e/users/73dc59c5ec454b93bdbca717d35cbb33/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:57.997114 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e8d414-b294-4f4f-8a73-e754914b43c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=744ac10864bf4318bc8ee17d436a0f5e, user_id=73dc59c5ec454b93bdbca717d35cbb33, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:58.001718 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e8d414-b294-4f4f-8a73-e754914b43c2 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:58.022405 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1173/2378] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:03:57 2022] PUT /identity/v3/projects/744ac10864bf4318bc8ee17d436a0f5e/users/73dc59c5ec454b93bdbca717d35cbb33/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:58.026269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9578f32-5ca1-4dd4-96f0-a017d09e9967 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:58.039017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9578f32-5ca1-4dd4-96f0-a017d09e9967 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:58.112880 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f027d5cf-eea8-489a-a745-7286c898547b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:58.113420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f027d5cf-eea8-489a-a745-7286c898547b admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:58.113810 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f027d5cf-eea8-489a-a745-7286c898547b admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:58.114268 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f027d5cf-eea8-489a-a745-7286c898547b admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:58.114826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f027d5cf-eea8-489a-a745-7286c898547b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:58.117715 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f027d5cf-eea8-489a-a745-7286c898547b admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:58.133056 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1206/2379] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:03:58 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:03:58.134238 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e8d414-b294-4f4f-8a73-e754914b43c2 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:58.149974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e8d414-b294-4f4f-8a73-e754914b43c2 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:58.155650 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b17729e-95cd-4651-bb24-2d92ba642f29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:58.156601 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b17729e-95cd-4651-bb24-2d92ba642f29 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:58.156981 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b17729e-95cd-4651-bb24-2d92ba642f29 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:58.157357 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b17729e-95cd-4651-bb24-2d92ba642f29 admin admin] PATH_INFO: `/v3/projects/744ac10864bf4318bc8ee17d436a0f5e/users/73dc59c5ec454b93bdbca717d35cbb33/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:58.159423 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b17729e-95cd-4651-bb24-2d92ba642f29 admin admin] RBAC: Authorizing `identity:create_grant(project_id=744ac10864bf4318bc8ee17d436a0f5e, user_id=73dc59c5ec454b93bdbca717d35cbb33, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:58.162856 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b17729e-95cd-4651-bb24-2d92ba642f29 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:58.179131 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1174/2380] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:03:58 2022] PUT /identity/v3/projects/744ac10864bf4318bc8ee17d436a0f5e/users/73dc59c5ec454b93bdbca717d35cbb33/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:03:58.189788 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e446757e-5426-4097-9bf1-968fae24de2f None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:58.190298 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e446757e-5426-4097-9bf1-968fae24de2f None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:58.190773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e446757e-5426-4097-9bf1-968fae24de2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:58.224653 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-e446757e-5426-4097-9bf1-968fae24de2f None None] MFA Rules not processed for user `73dc59c5ec454b93bdbca717d35cbb33`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:58.225882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e446757e-5426-4097-9bf1-968fae24de2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:58.314362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1207/2381] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:58 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:58.328748 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b17729e-95cd-4651-bb24-2d92ba642f29 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:58.331062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-1b17729e-95cd-4651-bb24-2d92ba642f29 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:58.405481 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b17729e-95cd-4651-bb24-2d92ba642f29 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:58.409827 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2e59db-8bd1-4724-82c6-620a6c7b09a4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:58.412030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2e59db-8bd1-4724-82c6-620a6c7b09a4 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:58.415800 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2e59db-8bd1-4724-82c6-620a6c7b09a4 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:58.416030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2e59db-8bd1-4724-82c6-620a6c7b09a4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:58.428744 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2e59db-8bd1-4724-82c6-620a6c7b09a4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:58.430715 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2e59db-8bd1-4724-82c6-620a6c7b09a4 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:58.443246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1175/2382] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:58 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 15:03:58.479260 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e446757e-5426-4097-9bf1-968fae24de2f None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:58.547859 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:58.563532 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e446757e-5426-4097-9bf1-968fae24de2f None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:58.572851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5de407-239e-4170-9879-27233fd24d47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:58.573656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5de407-239e-4170-9879-27233fd24d47 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:58.573942 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5de407-239e-4170-9879-27233fd24d47 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:58.574220 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5de407-239e-4170-9879-27233fd24d47 admin admin] PATH_INFO: `/v3/users/ec8a3ee7912c4bd1bea33d1628d59981` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:58.575734 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5de407-239e-4170-9879-27233fd24d47 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec8a3ee7912c4bd1bea33d1628d59981)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:58.578676 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5de407-239e-4170-9879-27233fd24d47 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:58.646146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8b5de407-239e-4170-9879-27233fd24d47 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec8a3ee7912c4bd1bea33d1628d59981'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:58.646817 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8b5de407-239e-4170-9879-27233fd24d47 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec8a3ee7912c4bd1bea33d1628d59981'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:58.659879 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8b5de407-239e-4170-9879-27233fd24d47 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec8a3ee7912c4bd1bea33d1628d59981'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:58.669242 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8b5de407-239e-4170-9879-27233fd24d47 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec8a3ee7912c4bd1bea33d1628d59981'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:58.678507 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-8b5de407-239e-4170-9879-27233fd24d47 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec8a3ee7912c4bd1bea33d1628d59981'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:58.685992 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1208/2383] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:03:58 2022] DELETE /identity/v3/users/ec8a3ee7912c4bd1bea33d1628d59981 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:58.690412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc882cea-aad2-4baa-bdb6-2a16715d1dd9 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:58.690708 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc882cea-aad2-4baa-bdb6-2a16715d1dd9 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:58.690986 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc882cea-aad2-4baa-bdb6-2a16715d1dd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:58.714963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-fc882cea-aad2-4baa-bdb6-2a16715d1dd9 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:03:58.716236 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-fc882cea-aad2-4baa-bdb6-2a16715d1dd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:58.791238 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1176/2384] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:03:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:03:58.801761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5de407-239e-4170-9879-27233fd24d47 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:58.804051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-8b5de407-239e-4170-9879-27233fd24d47 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:58.882877 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5de407-239e-4170-9879-27233fd24d47 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:58.888214 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a5d8df-3ed3-4b00-aec0-eff874cc4e73 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:58.889107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a5d8df-3ed3-4b00-aec0-eff874cc4e73 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:58.889484 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a5d8df-3ed3-4b00-aec0-eff874cc4e73 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:58.889853 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a5d8df-3ed3-4b00-aec0-eff874cc4e73 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:58.904790 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a5d8df-3ed3-4b00-aec0-eff874cc4e73 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:58.909277 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a5d8df-3ed3-4b00-aec0-eff874cc4e73 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:58.924435 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1209/2385] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:03:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:03:59.218491 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc882cea-aad2-4baa-bdb6-2a16715d1dd9 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:59.220105 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-fc882cea-aad2-4baa-bdb6-2a16715d1dd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:59.237477 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc882cea-aad2-4baa-bdb6-2a16715d1dd9 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:59.243406 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fe8957-50f5-464c-902b-19ccc2cb1fe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:59.244083 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fe8957-50f5-464c-902b-19ccc2cb1fe0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:59.244385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fe8957-50f5-464c-902b-19ccc2cb1fe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:59.244638 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fe8957-50f5-464c-902b-19ccc2cb1fe0 admin admin] PATH_INFO: `/v3/projects/001f1cca96b243ab9fb2fe8c555ada77` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:59.245767 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fe8957-50f5-464c-902b-19ccc2cb1fe0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=001f1cca96b243ab9fb2fe8c555ada77)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:59.247711 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fe8957-50f5-464c-902b-19ccc2cb1fe0 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:59.297176 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-21fe8957-50f5-464c-902b-19ccc2cb1fe0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '001f1cca96b243ab9fb2fe8c555ada77', 'request_id': 'req-21fe8957-50f5-464c-902b-19ccc2cb1fe0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:59.299108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:INSERT=5,keystone:SELECT=91,keystone:DELETE=4 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:59.308059 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-21fe8957-50f5-464c-902b-19ccc2cb1fe0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '001f1cca96b243ab9fb2fe8c555ada77', 'request_id': 'req-21fe8957-50f5-464c-902b-19ccc2cb1fe0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:59.313875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-21fe8957-50f5-464c-902b-19ccc2cb1fe0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '001f1cca96b243ab9fb2fe8c555ada77', 'request_id': 'req-21fe8957-50f5-464c-902b-19ccc2cb1fe0'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:59.327317 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-21fe8957-50f5-464c-902b-19ccc2cb1fe0 admin admin] The token cache is being invalidate because project 001f1cca96b243ab9fb2fe8c555ada77 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:03:59.327756 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-21fe8957-50f5-464c-902b-19ccc2cb1fe0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:59.328294 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-21fe8957-50f5-464c-902b-19ccc2cb1fe0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:59.329817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1177/2386] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:03:59 2022] DELETE /identity/v3/projects/001f1cca96b243ab9fb2fe8c555ada77 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:59.334121 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a5d8df-3ed3-4b00-aec0-eff874cc4e73 service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:59.344387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a5d8df-3ed3-4b00-aec0-eff874cc4e73 service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:59.408396 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1dba105-b17a-4e0b-b33f-968c59852337 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:59.409374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dba105-b17a-4e0b-b33f-968c59852337 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:59.409374 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dba105-b17a-4e0b-b33f-968c59852337 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:59.409625 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dba105-b17a-4e0b-b33f-968c59852337 admin admin] PATH_INFO: `/v3/users/73dc59c5ec454b93bdbca717d35cbb33` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:59.411360 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1dba105-b17a-4e0b-b33f-968c59852337 admin admin] RBAC: Authorizing `identity:delete_user(user_id=73dc59c5ec454b93bdbca717d35cbb33)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:59.413405 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1dba105-b17a-4e0b-b33f-968c59852337 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:03:59.431385 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:03:59.464240 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b1dba105-b17a-4e0b-b33f-968c59852337 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '73dc59c5ec454b93bdbca717d35cbb33'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:59.464900 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b1dba105-b17a-4e0b-b33f-968c59852337 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '73dc59c5ec454b93bdbca717d35cbb33'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:59.478580 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b1dba105-b17a-4e0b-b33f-968c59852337 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '73dc59c5ec454b93bdbca717d35cbb33'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:59.485466 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b1dba105-b17a-4e0b-b33f-968c59852337 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '73dc59c5ec454b93bdbca717d35cbb33'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:59.494168 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-b1dba105-b17a-4e0b-b33f-968c59852337 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '73dc59c5ec454b93bdbca717d35cbb33'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:03:59.511659 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1210/2387] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:03:59 2022] DELETE /identity/v3/users/73dc59c5ec454b93bdbca717d35cbb33 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:03:59.856316 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fe8957-50f5-464c-902b-19ccc2cb1fe0 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:03:59.858103 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-21fe8957-50f5-464c-902b-19ccc2cb1fe0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:03:59.884177 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fe8957-50f5-464c-902b-19ccc2cb1fe0 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:03:59.989086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0ac788-6e9a-4bf1-bd94-e86aaf8482d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:03:59.990164 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0ac788-6e9a-4bf1-bd94-e86aaf8482d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:03:59.990872 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0ac788-6e9a-4bf1-bd94-e86aaf8482d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:03:59.991218 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0ac788-6e9a-4bf1-bd94-e86aaf8482d8 admin admin] PATH_INFO: `/v3/projects/744ac10864bf4318bc8ee17d436a0f5e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:03:59.992524 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0ac788-6e9a-4bf1-bd94-e86aaf8482d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=744ac10864bf4318bc8ee17d436a0f5e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:03:59.999746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0ac788-6e9a-4bf1-bd94-e86aaf8482d8 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:00.062778 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5d0ac788-6e9a-4bf1-bd94-e86aaf8482d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '744ac10864bf4318bc8ee17d436a0f5e', 'request_id': 'req-5d0ac788-6e9a-4bf1-bd94-e86aaf8482d8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:00.074232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5d0ac788-6e9a-4bf1-bd94-e86aaf8482d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '744ac10864bf4318bc8ee17d436a0f5e', 'request_id': 'req-5d0ac788-6e9a-4bf1-bd94-e86aaf8482d8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:00.082817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5d0ac788-6e9a-4bf1-bd94-e86aaf8482d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '744ac10864bf4318bc8ee17d436a0f5e', 'request_id': 'req-5d0ac788-6e9a-4bf1-bd94-e86aaf8482d8'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:00.096427 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5d0ac788-6e9a-4bf1-bd94-e86aaf8482d8 admin admin] The token cache is being invalidate because project 744ac10864bf4318bc8ee17d436a0f5e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:04:00.096737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5d0ac788-6e9a-4bf1-bd94-e86aaf8482d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:00.097412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5d0ac788-6e9a-4bf1-bd94-e86aaf8482d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:00.099167 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1178/2388] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:03:59 2022] DELETE /identity/v3/projects/744ac10864bf4318bc8ee17d436a0f5e => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:00.104214 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7134ff98-7757-4486-b0e5-4d25a1d008fb None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:00.104543 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7134ff98-7757-4486-b0e5-4d25a1d008fb None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:00.105627 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7134ff98-7757-4486-b0e5-4d25a1d008fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:00.143387 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-7134ff98-7757-4486-b0e5-4d25a1d008fb None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:00.144864 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7134ff98-7757-4486-b0e5-4d25a1d008fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:00.213217 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1211/2389] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:00.221518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66be31b-bec3-4869-b4cd-1cf3fff06f05 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:00.221972 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66be31b-bec3-4869-b4cd-1cf3fff06f05 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:00.222138 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66be31b-bec3-4869-b4cd-1cf3fff06f05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:00.251318 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-f66be31b-bec3-4869-b4cd-1cf3fff06f05 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:00.253010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-f66be31b-bec3-4869-b4cd-1cf3fff06f05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:00.264229 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1179/2390] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:00.268713 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7134ff98-7757-4486-b0e5-4d25a1d008fb None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:00.279725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7134ff98-7757-4486-b0e5-4d25a1d008fb None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:00.285494 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff146ac-2cf2-44d6-ae56-8ede9725daf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:00.286391 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff146ac-2cf2-44d6-ae56-8ede9725daf7 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:00.286634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff146ac-2cf2-44d6-ae56-8ede9725daf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:00.287056 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff146ac-2cf2-44d6-ae56-8ede9725daf7 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:00.287977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff146ac-2cf2-44d6-ae56-8ede9725daf7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:04:00.288285 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff146ac-2cf2-44d6-ae56-8ede9725daf7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:00.289954 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff146ac-2cf2-44d6-ae56-8ede9725daf7 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:00.303588 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1212/2391] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:04:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:04:00.308012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66be31b-bec3-4869-b4cd-1cf3fff06f05 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:00.319019 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66be31b-bec3-4869-b4cd-1cf3fff06f05 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:00.324466 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d681cd-1c16-48ca-924b-e70f4ac32a0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:00.325193 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d681cd-1c16-48ca-924b-e70f4ac32a0a admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:00.325437 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d681cd-1c16-48ca-924b-e70f4ac32a0a admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:00.325755 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d681cd-1c16-48ca-924b-e70f4ac32a0a admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:00.326411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d681cd-1c16-48ca-924b-e70f4ac32a0a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:00.328399 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d681cd-1c16-48ca-924b-e70f4ac32a0a admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:00.359104 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1180/2392] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:04:00 2022] POST /identity/v3/projects => generated 381 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:04:00.360274 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff146ac-2cf2-44d6-ae56-8ede9725daf7 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:00.373214 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff146ac-2cf2-44d6-ae56-8ede9725daf7 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:00.437201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15e0a22-cb34-4533-9648-4547b4ec0bdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:00.437934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15e0a22-cb34-4533-9648-4547b4ec0bdd admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:00.438293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15e0a22-cb34-4533-9648-4547b4ec0bdd admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:00.438570 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15e0a22-cb34-4533-9648-4547b4ec0bdd admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:00.439223 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15e0a22-cb34-4533-9648-4547b4ec0bdd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:00.441536 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15e0a22-cb34-4533-9648-4547b4ec0bdd admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:00.466925 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1213/2393] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:04:00 2022] POST /identity/v3/users => generated 291 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:04:00.471045 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d681cd-1c16-48ca-924b-e70f4ac32a0a admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:00.481906 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d681cd-1c16-48ca-924b-e70f4ac32a0a admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:00.487857 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d642c6-fec5-4562-a450-e1bdcf749af9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:00.488538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d642c6-fec5-4562-a450-e1bdcf749af9 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:00.489381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d642c6-fec5-4562-a450-e1bdcf749af9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:00.489784 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d642c6-fec5-4562-a450-e1bdcf749af9 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:00.490609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d642c6-fec5-4562-a450-e1bdcf749af9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:00.492445 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d642c6-fec5-4562-a450-e1bdcf749af9 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:00.503341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1181/2394] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:00 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:04:00.507604 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15e0a22-cb34-4533-9648-4547b4ec0bdd admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:00.517668 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15e0a22-cb34-4533-9648-4547b4ec0bdd admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:00.522452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f349ff0b-1179-4b3a-9ce7-14420a10b97d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:00.523102 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f349ff0b-1179-4b3a-9ce7-14420a10b97d admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:00.523383 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f349ff0b-1179-4b3a-9ce7-14420a10b97d admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:00.523692 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f349ff0b-1179-4b3a-9ce7-14420a10b97d admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:00.524335 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f349ff0b-1179-4b3a-9ce7-14420a10b97d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:00.525918 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f349ff0b-1179-4b3a-9ce7-14420a10b97d admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:00.531344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:00.538632 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1214/2395] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:00 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:04:00.543101 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d642c6-fec5-4562-a450-e1bdcf749af9 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:00.554471 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d642c6-fec5-4562-a450-e1bdcf749af9 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:00.559353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05951d0-21d2-4650-9318-413a04884466 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:00.560064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05951d0-21d2-4650-9318-413a04884466 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:00.560346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05951d0-21d2-4650-9318-413a04884466 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:00.560626 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05951d0-21d2-4650-9318-413a04884466 admin admin] PATH_INFO: `/v3/projects/6d0bcd5c3a3646028b651f2ca7d6c27a/users/05c2bd0fbaff434ea14907459c0b039b/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:00.576872 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05951d0-21d2-4650-9318-413a04884466 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d0bcd5c3a3646028b651f2ca7d6c27a, user_id=05c2bd0fbaff434ea14907459c0b039b, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:00.579545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05951d0-21d2-4650-9318-413a04884466 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:00.592570 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1182/2396] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:04:00 2022] PUT /identity/v3/projects/6d0bcd5c3a3646028b651f2ca7d6c27a/users/05c2bd0fbaff434ea14907459c0b039b/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:00.603495 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031a351e-0986-4424-a7e1-a1cbb265a7b1 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:00.603801 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031a351e-0986-4424-a7e1-a1cbb265a7b1 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:00.604076 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031a351e-0986-4424-a7e1-a1cbb265a7b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:00.634023 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-031a351e-0986-4424-a7e1-a1cbb265a7b1 None None] MFA Rules not processed for user `05c2bd0fbaff434ea14907459c0b039b`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:00.635256 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-031a351e-0986-4424-a7e1-a1cbb265a7b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:00.702038 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1215/2397] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:00 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:00.707026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05951d0-21d2-4650-9318-413a04884466 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:00.712820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:INSERT=4,keystone:DELETE=10,keystone:SELECT=86 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:00.776576 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05951d0-21d2-4650-9318-413a04884466 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:00.781286 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e873ed06-26f7-4468-8fab-7fa189e596a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:00.781891 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e873ed06-26f7-4468-8fab-7fa189e596a0 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:00.782177 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e873ed06-26f7-4468-8fab-7fa189e596a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:00.782456 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e873ed06-26f7-4468-8fab-7fa189e596a0 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:00.783087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e873ed06-26f7-4468-8fab-7fa189e596a0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:00.785029 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e873ed06-26f7-4468-8fab-7fa189e596a0 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:00.815418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1183/2398] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:04:00 2022] POST /identity/v3/projects => generated 381 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:04:00.819874 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031a351e-0986-4424-a7e1-a1cbb265a7b1 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:00.830817 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031a351e-0986-4424-a7e1-a1cbb265a7b1 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:00.891235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832d28ef-b913-4b0c-a152-6be3d6daaf06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:00.891882 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832d28ef-b913-4b0c-a152-6be3d6daaf06 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:00.892167 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832d28ef-b913-4b0c-a152-6be3d6daaf06 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:00.892444 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832d28ef-b913-4b0c-a152-6be3d6daaf06 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:00.893084 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832d28ef-b913-4b0c-a152-6be3d6daaf06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:00.895492 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832d28ef-b913-4b0c-a152-6be3d6daaf06 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:00.918549 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1216/2399] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:04:00 2022] POST /identity/v3/users => generated 297 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:04:00.923251 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e873ed06-26f7-4468-8fab-7fa189e596a0 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:00.935952 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e873ed06-26f7-4468-8fab-7fa189e596a0 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:00.940818 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0362bdbd-3d9f-4a66-8bd2-a460c1b1e378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:00.941453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0362bdbd-3d9f-4a66-8bd2-a460c1b1e378 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:00.942407 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0362bdbd-3d9f-4a66-8bd2-a460c1b1e378 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:00.942678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0362bdbd-3d9f-4a66-8bd2-a460c1b1e378 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:00.943326 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0362bdbd-3d9f-4a66-8bd2-a460c1b1e378 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:00.945233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0362bdbd-3d9f-4a66-8bd2-a460c1b1e378 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:00.956142 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1184/2400] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:00 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:04:00.960862 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832d28ef-b913-4b0c-a152-6be3d6daaf06 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:00.971058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832d28ef-b913-4b0c-a152-6be3d6daaf06 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:00.976145 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b00827-1934-44a7-9d73-567677982f32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:00.976798 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b00827-1934-44a7-9d73-567677982f32 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:00.977080 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b00827-1934-44a7-9d73-567677982f32 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:00.977354 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b00827-1934-44a7-9d73-567677982f32 admin admin] PATH_INFO: `/v3/projects/7a96d9279c2f472a91c662711c3e242b/users/db5088ec55d548408f781a400518cd1c/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:00.998645 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b00827-1934-44a7-9d73-567677982f32 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a96d9279c2f472a91c662711c3e242b, user_id=db5088ec55d548408f781a400518cd1c, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:01.001721 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b00827-1934-44a7-9d73-567677982f32 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:01.017683 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1217/2401] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:04:00 2022] PUT /identity/v3/projects/7a96d9279c2f472a91c662711c3e242b/users/db5088ec55d548408f781a400518cd1c/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:01.021851 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0362bdbd-3d9f-4a66-8bd2-a460c1b1e378 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:01.033223 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0362bdbd-3d9f-4a66-8bd2-a460c1b1e378 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:01.105202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5d726a-acf6-4b77-99f1-0495245fd969 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:01.105820 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5d726a-acf6-4b77-99f1-0495245fd969 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:01.106099 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5d726a-acf6-4b77-99f1-0495245fd969 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:01.106404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5d726a-acf6-4b77-99f1-0495245fd969 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:01.107051 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5d726a-acf6-4b77-99f1-0495245fd969 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:01.108688 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5d726a-acf6-4b77-99f1-0495245fd969 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:01.119607 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1185/2402] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:01 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:04:01.123599 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b00827-1934-44a7-9d73-567677982f32 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:01.135833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b00827-1934-44a7-9d73-567677982f32 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:01.140588 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0d66f4-450c-4cdd-a0ee-e147c06e82ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:01.141251 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0d66f4-450c-4cdd-a0ee-e147c06e82ff admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:01.141534 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0d66f4-450c-4cdd-a0ee-e147c06e82ff admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:01.141814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0d66f4-450c-4cdd-a0ee-e147c06e82ff admin admin] PATH_INFO: `/v3/projects/7a96d9279c2f472a91c662711c3e242b/users/db5088ec55d548408f781a400518cd1c/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:01.143784 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0d66f4-450c-4cdd-a0ee-e147c06e82ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a96d9279c2f472a91c662711c3e242b, user_id=db5088ec55d548408f781a400518cd1c, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:01.146604 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0d66f4-450c-4cdd-a0ee-e147c06e82ff admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:01.157292 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1218/2403] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:04:01 2022] PUT /identity/v3/projects/7a96d9279c2f472a91c662711c3e242b/users/db5088ec55d548408f781a400518cd1c/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:01.169740 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4804cc96-e45a-44f0-811a-9cb402d00cd3 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:01.170033 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4804cc96-e45a-44f0-811a-9cb402d00cd3 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:01.170305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4804cc96-e45a-44f0-811a-9cb402d00cd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:01.203700 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-4804cc96-e45a-44f0-811a-9cb402d00cd3 None None] MFA Rules not processed for user `db5088ec55d548408f781a400518cd1c`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:01.204960 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-4804cc96-e45a-44f0-811a-9cb402d00cd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:01.270891 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:01.275571 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1186/2404] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:01 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:01.286546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0d66f4-450c-4cdd-a0ee-e147c06e82ff admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:01.288292 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-de0d66f4-450c-4cdd-a0ee-e147c06e82ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:01.334606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:01.347084 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0d66f4-450c-4cdd-a0ee-e147c06e82ff admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:01.351519 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80859fa-a492-4773-9907-282a33ce9ab0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:01.352161 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80859fa-a492-4773-9907-282a33ce9ab0 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:01.352469 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80859fa-a492-4773-9907-282a33ce9ab0 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:01.352749 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80859fa-a492-4773-9907-282a33ce9ab0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:01.362333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80859fa-a492-4773-9907-282a33ce9ab0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:01.364557 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80859fa-a492-4773-9907-282a33ce9ab0 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:01.374367 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1219/2405] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:01 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:01.490134 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4804cc96-e45a-44f0-811a-9cb402d00cd3 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:01.500230 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4804cc96-e45a-44f0-811a-9cb402d00cd3 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:01.504420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64aa1fe3-3532-4817-9431-0831c0c3a290 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:01.505220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64aa1fe3-3532-4817-9431-0831c0c3a290 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:01.505498 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64aa1fe3-3532-4817-9431-0831c0c3a290 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:01.505768 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64aa1fe3-3532-4817-9431-0831c0c3a290 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:01.577623 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64aa1fe3-3532-4817-9431-0831c0c3a290 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:01.579492 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64aa1fe3-3532-4817-9431-0831c0c3a290 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:01.591793 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1187/2406] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:01 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:03.233196 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80859fa-a492-4773-9907-282a33ce9ab0 service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:03.234600 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d80859fa-a492-4773-9907-282a33ce9ab0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:03.293653 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80859fa-a492-4773-9907-282a33ce9ab0 service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:03.297269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46ab347-a99c-443a-b1d0-a2f1dd1d4101 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:03.297868 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46ab347-a99c-443a-b1d0-a2f1dd1d4101 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:03.298146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46ab347-a99c-443a-b1d0-a2f1dd1d4101 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:03.298420 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46ab347-a99c-443a-b1d0-a2f1dd1d4101 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:03.300169 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b46ab347-a99c-443a-b1d0-a2f1dd1d4101 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:03.310352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46ab347-a99c-443a-b1d0-a2f1dd1d4101 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:03.311826 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46ab347-a99c-443a-b1d0-a2f1dd1d4101 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:03.321257 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1220/2407] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:03 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:04.273741 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64aa1fe3-3532-4817-9431-0831c0c3a290 service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:04.348495 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64aa1fe3-3532-4817-9431-0831c0c3a290 service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:04.353101 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b7f4f3-a5ce-401e-91bf-4cdf60cfc1cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:04.353754 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b7f4f3-a5ce-401e-91bf-4cdf60cfc1cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:04.354002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b7f4f3-a5ce-401e-91bf-4cdf60cfc1cf admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:04.354277 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b7f4f3-a5ce-401e-91bf-4cdf60cfc1cf admin admin] PATH_INFO: `/v3/users/05c2bd0fbaff434ea14907459c0b039b` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:04.355431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b7f4f3-a5ce-401e-91bf-4cdf60cfc1cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=05c2bd0fbaff434ea14907459c0b039b)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:04.357508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b7f4f3-a5ce-401e-91bf-4cdf60cfc1cf admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:04.412734 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-41b7f4f3-a5ce-401e-91bf-4cdf60cfc1cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05c2bd0fbaff434ea14907459c0b039b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:04.419760 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-41b7f4f3-a5ce-401e-91bf-4cdf60cfc1cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05c2bd0fbaff434ea14907459c0b039b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:04.420444 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-41b7f4f3-a5ce-401e-91bf-4cdf60cfc1cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05c2bd0fbaff434ea14907459c0b039b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:04.421051 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-41b7f4f3-a5ce-401e-91bf-4cdf60cfc1cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05c2bd0fbaff434ea14907459c0b039b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:04.434771 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-41b7f4f3-a5ce-401e-91bf-4cdf60cfc1cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05c2bd0fbaff434ea14907459c0b039b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:04.448717 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1188/2408] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:04:04 2022] DELETE /identity/v3/users/05c2bd0fbaff434ea14907459c0b039b => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:04.453518 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d74848-2ae3-4312-9808-8b596ba5cabb None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:04.454017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d74848-2ae3-4312-9808-8b596ba5cabb None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:04.454113 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d74848-2ae3-4312-9808-8b596ba5cabb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:04.477567 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-a9d74848-2ae3-4312-9808-8b596ba5cabb None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:04.478845 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a9d74848-2ae3-4312-9808-8b596ba5cabb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:04.541653 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1221/2409] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:04.552042 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b7f4f3-a5ce-401e-91bf-4cdf60cfc1cf admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:04.553449 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-41b7f4f3-a5ce-401e-91bf-4cdf60cfc1cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:04.574178 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:04.612562 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b7f4f3-a5ce-401e-91bf-4cdf60cfc1cf admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:04.617075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e1bb6d-f738-49b6-acf3-42cf1cda4b83 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:04.617695 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e1bb6d-f738-49b6-acf3-42cf1cda4b83 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:04.618171 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e1bb6d-f738-49b6-acf3-42cf1cda4b83 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:04.618528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e1bb6d-f738-49b6-acf3-42cf1cda4b83 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:04.629691 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e1bb6d-f738-49b6-acf3-42cf1cda4b83 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:04.631281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e1bb6d-f738-49b6-acf3-42cf1cda4b83 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:04.640826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1189/2410] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:05.073320 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d74848-2ae3-4312-9808-8b596ba5cabb None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:05.074746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a9d74848-2ae3-4312-9808-8b596ba5cabb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:05.087736 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d74848-2ae3-4312-9808-8b596ba5cabb None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:05.100399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f111cf-c654-4d3f-8016-115511a04988 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:05.102007 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f111cf-c654-4d3f-8016-115511a04988 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:05.103408 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f111cf-c654-4d3f-8016-115511a04988 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:05.104418 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f111cf-c654-4d3f-8016-115511a04988 admin admin] PATH_INFO: `/v3/projects/6d0bcd5c3a3646028b651f2ca7d6c27a` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:05.108437 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f111cf-c654-4d3f-8016-115511a04988 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d0bcd5c3a3646028b651f2ca7d6c27a)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:05.111688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f111cf-c654-4d3f-8016-115511a04988 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:05.180959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-d9f111cf-c654-4d3f-8016-115511a04988 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d0bcd5c3a3646028b651f2ca7d6c27a', 'request_id': 'req-d9f111cf-c654-4d3f-8016-115511a04988'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:05.187096 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-d9f111cf-c654-4d3f-8016-115511a04988 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d0bcd5c3a3646028b651f2ca7d6c27a', 'request_id': 'req-d9f111cf-c654-4d3f-8016-115511a04988'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:05.194119 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-d9f111cf-c654-4d3f-8016-115511a04988 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d0bcd5c3a3646028b651f2ca7d6c27a', 'request_id': 'req-d9f111cf-c654-4d3f-8016-115511a04988'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:05.208848 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-d9f111cf-c654-4d3f-8016-115511a04988 admin admin] The token cache is being invalidate because project 6d0bcd5c3a3646028b651f2ca7d6c27a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:04:05.209386 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-d9f111cf-c654-4d3f-8016-115511a04988 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:05.211102 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-d9f111cf-c654-4d3f-8016-115511a04988 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:05.212700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1222/2411] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:04:05 2022] DELETE /identity/v3/projects/6d0bcd5c3a3646028b651f2ca7d6c27a => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:05.216925 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e1bb6d-f738-49b6-acf3-42cf1cda4b83 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:05.228108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e1bb6d-f738-49b6-acf3-42cf1cda4b83 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:05.304861 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63a1692-7a70-43c4-a091-0f1104255fc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:05.305567 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63a1692-7a70-43c4-a091-0f1104255fc2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:05.305855 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63a1692-7a70-43c4-a091-0f1104255fc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:05.306264 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63a1692-7a70-43c4-a091-0f1104255fc2 admin admin] PATH_INFO: `/v3/users/db5088ec55d548408f781a400518cd1c` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:05.307412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63a1692-7a70-43c4-a091-0f1104255fc2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=db5088ec55d548408f781a400518cd1c)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:05.309402 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63a1692-7a70-43c4-a091-0f1104255fc2 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:05.371268 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b63a1692-7a70-43c4-a091-0f1104255fc2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'db5088ec55d548408f781a400518cd1c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:05.379833 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b63a1692-7a70-43c4-a091-0f1104255fc2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'db5088ec55d548408f781a400518cd1c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:05.380595 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b63a1692-7a70-43c4-a091-0f1104255fc2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'db5088ec55d548408f781a400518cd1c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:05.381318 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b63a1692-7a70-43c4-a091-0f1104255fc2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'db5088ec55d548408f781a400518cd1c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:05.396439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b63a1692-7a70-43c4-a091-0f1104255fc2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'db5088ec55d548408f781a400518cd1c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:05.414728 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1190/2412] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:04:05 2022] DELETE /identity/v3/users/db5088ec55d548408f781a400518cd1c => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:05.734523 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f111cf-c654-4d3f-8016-115511a04988 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:05.736730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d9f111cf-c654-4d3f-8016-115511a04988 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:05.760251 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f111cf-c654-4d3f-8016-115511a04988 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:05.771071 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:05.833309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bfcfae-c307-4283-b47c-01f0408a97ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:05.833977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bfcfae-c307-4283-b47c-01f0408a97ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:05.834398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bfcfae-c307-4283-b47c-01f0408a97ee admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:05.834691 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bfcfae-c307-4283-b47c-01f0408a97ee admin admin] PATH_INFO: `/v3/projects/7a96d9279c2f472a91c662711c3e242b` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:05.835890 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bfcfae-c307-4283-b47c-01f0408a97ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a96d9279c2f472a91c662711c3e242b)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:05.838017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bfcfae-c307-4283-b47c-01f0408a97ee admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:05.904089 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e7bfcfae-c307-4283-b47c-01f0408a97ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a96d9279c2f472a91c662711c3e242b', 'request_id': 'req-e7bfcfae-c307-4283-b47c-01f0408a97ee'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:05.915389 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e7bfcfae-c307-4283-b47c-01f0408a97ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a96d9279c2f472a91c662711c3e242b', 'request_id': 'req-e7bfcfae-c307-4283-b47c-01f0408a97ee'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:05.922325 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e7bfcfae-c307-4283-b47c-01f0408a97ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a96d9279c2f472a91c662711c3e242b', 'request_id': 'req-e7bfcfae-c307-4283-b47c-01f0408a97ee'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:05.941871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e7bfcfae-c307-4283-b47c-01f0408a97ee admin admin] The token cache is being invalidate because project 7a96d9279c2f472a91c662711c3e242b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:04:05.942174 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e7bfcfae-c307-4283-b47c-01f0408a97ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:05.942871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-e7bfcfae-c307-4283-b47c-01f0408a97ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:05.944903 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1223/2413] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:04:05 2022] DELETE /identity/v3/projects/7a96d9279c2f472a91c662711c3e242b => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:05.950231 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a688bef-4839-44e0-8b8a-40119d83c34b None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:05.950704 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a688bef-4839-44e0-8b8a-40119d83c34b None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:05.951084 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a688bef-4839-44e0-8b8a-40119d83c34b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:05.976680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-0a688bef-4839-44e0-8b8a-40119d83c34b None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:05.977929 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-0a688bef-4839-44e0-8b8a-40119d83c34b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:06.024815 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:06.078002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1191/2414] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:06.082763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bfcfae-c307-4283-b47c-01f0408a97ee admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:06.083827 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbad3c1b-3e8b-45f5-810d-32ffb101eb41 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:06.083827 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbad3c1b-3e8b-45f5-810d-32ffb101eb41 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:06.083827 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbad3c1b-3e8b-45f5-810d-32ffb101eb41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:06.091756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-e7bfcfae-c307-4283-b47c-01f0408a97ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:06.112026 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bfcfae-c307-4283-b47c-01f0408a97ee admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:06.112426 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-dbad3c1b-3e8b-45f5-810d-32ffb101eb41 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:06.113915 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-dbad3c1b-3e8b-45f5-810d-32ffb101eb41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:06.118530 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6765975e-2e8d-41da-89c0-20bbc6379d2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:06.119372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6765975e-2e8d-41da-89c0-20bbc6379d2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:06.119860 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6765975e-2e8d-41da-89c0-20bbc6379d2c admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:06.120575 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6765975e-2e8d-41da-89c0-20bbc6379d2c admin admin] PATH_INFO: `/v3/users/123f2d4be7324f4c9a336e8f6c3db951` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:06.122365 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6765975e-2e8d-41da-89c0-20bbc6379d2c admin admin] RBAC: Authorizing `identity:delete_user(user_id=123f2d4be7324f4c9a336e8f6c3db951)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:06.125129 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6765975e-2e8d-41da-89c0-20bbc6379d2c admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:06.127031 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1192/2415] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:06.132939 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbad3c1b-3e8b-45f5-810d-32ffb101eb41 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:06.147716 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbad3c1b-3e8b-45f5-810d-32ffb101eb41 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:06.159264 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ca14fd-a140-437f-bca3-2dc95ef92872 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:06.159929 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ca14fd-a140-437f-bca3-2dc95ef92872 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:06.160945 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ca14fd-a140-437f-bca3-2dc95ef92872 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:06.161237 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ca14fd-a140-437f-bca3-2dc95ef92872 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:06.161905 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ca14fd-a140-437f-bca3-2dc95ef92872 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:04:06.162375 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ca14fd-a140-437f-bca3-2dc95ef92872 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:06.164257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ca14fd-a140-437f-bca3-2dc95ef92872 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:06.184813 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1193/2416] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:04:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:04:06.186095 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ca14fd-a140-437f-bca3-2dc95ef92872 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:06.197112 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ca14fd-a140-437f-bca3-2dc95ef92872 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:06.203397 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b95e0a-f7d4-4105-842e-7272b83ad00c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:06.204504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b95e0a-f7d4-4105-842e-7272b83ad00c admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:06.204942 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b95e0a-f7d4-4105-842e-7272b83ad00c admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:06.205380 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b95e0a-f7d4-4105-842e-7272b83ad00c admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:06.206169 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b95e0a-f7d4-4105-842e-7272b83ad00c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:06.209076 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b95e0a-f7d4-4105-842e-7272b83ad00c admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:06.211570 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-6765975e-2e8d-41da-89c0-20bbc6379d2c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '123f2d4be7324f4c9a336e8f6c3db951'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:06.212368 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-6765975e-2e8d-41da-89c0-20bbc6379d2c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '123f2d4be7324f4c9a336e8f6c3db951'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:06.234101 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-6765975e-2e8d-41da-89c0-20bbc6379d2c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '123f2d4be7324f4c9a336e8f6c3db951'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:06.242289 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-6765975e-2e8d-41da-89c0-20bbc6379d2c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '123f2d4be7324f4c9a336e8f6c3db951'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:06.252435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1194/2417] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:04:06 2022] POST /identity/v3/projects => generated 385 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:04:06.256745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b95e0a-f7d4-4105-842e-7272b83ad00c admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:06.263910 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-6765975e-2e8d-41da-89c0-20bbc6379d2c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '123f2d4be7324f4c9a336e8f6c3db951'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:06.272117 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-07b95e0a-f7d4-4105-842e-7272b83ad00c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:06.273960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1224/2418] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:04:06 2022] DELETE /identity/v3/users/123f2d4be7324f4c9a336e8f6c3db951 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:06.278879 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07833ef-2d5a-47b9-9b26-5ee386a8d2bb None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:06.279174 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07833ef-2d5a-47b9-9b26-5ee386a8d2bb None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:06.279938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07833ef-2d5a-47b9-9b26-5ee386a8d2bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:06.304673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-b07833ef-2d5a-47b9-9b26-5ee386a8d2bb None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:06.306029 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b07833ef-2d5a-47b9-9b26-5ee386a8d2bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:06.316849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:06.366239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b95e0a-f7d4-4105-842e-7272b83ad00c admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:06.419723 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1225/2419] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:04:06.430561 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a76f4b7-036a-4b66-9365-3130f2d20966 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:06.431174 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a76f4b7-036a-4b66-9365-3130f2d20966 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:06.431309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07833ef-2d5a-47b9-9b26-5ee386a8d2bb None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:06.431451 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a76f4b7-036a-4b66-9365-3130f2d20966 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:06.431934 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a76f4b7-036a-4b66-9365-3130f2d20966 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:06.432746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b07833ef-2d5a-47b9-9b26-5ee386a8d2bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:06.432865 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a76f4b7-036a-4b66-9365-3130f2d20966 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:06.434845 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a76f4b7-036a-4b66-9365-3130f2d20966 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:06.461145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1195/2420] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:04:06 2022] POST /identity/v3/users => generated 293 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:04:06.465517 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a76f4b7-036a-4b66-9365-3130f2d20966 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:06.477210 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a76f4b7-036a-4b66-9365-3130f2d20966 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:06.483319 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8904bf5f-558d-411d-906f-334761ee3eaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:06.483983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8904bf5f-558d-411d-906f-334761ee3eaa admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:06.484287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8904bf5f-558d-411d-906f-334761ee3eaa admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:06.484563 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8904bf5f-558d-411d-906f-334761ee3eaa admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:06.485218 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8904bf5f-558d-411d-906f-334761ee3eaa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:06.486900 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8904bf5f-558d-411d-906f-334761ee3eaa admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:06.497712 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1196/2421] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:06 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:04:06.501810 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8904bf5f-558d-411d-906f-334761ee3eaa admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:06.513025 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07833ef-2d5a-47b9-9b26-5ee386a8d2bb None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:06.515366 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8904bf5f-558d-411d-906f-334761ee3eaa admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:06.517594 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078e0ace-8d46-47a4-a5b1-7681f635ad55 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:06.521384 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29765170-b9cc-416a-944c-393c19e88e2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:06.522232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29765170-b9cc-416a-944c-393c19e88e2e admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:06.522677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29765170-b9cc-416a-944c-393c19e88e2e admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:06.523143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29765170-b9cc-416a-944c-393c19e88e2e admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:06.524014 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29765170-b9cc-416a-944c-393c19e88e2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:06.524471 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078e0ace-8d46-47a4-a5b1-7681f635ad55 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:06.524992 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078e0ace-8d46-47a4-a5b1-7681f635ad55 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:06.525543 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078e0ace-8d46-47a4-a5b1-7681f635ad55 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:06.526946 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29765170-b9cc-416a-944c-393c19e88e2e admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:06.538774 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078e0ace-8d46-47a4-a5b1-7681f635ad55 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:06.540666 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078e0ace-8d46-47a4-a5b1-7681f635ad55 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:06.541766 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1197/2422] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:06 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:04:06.548109 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29765170-b9cc-416a-944c-393c19e88e2e admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:06.554413 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1226/2423] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 15:04:06.561874 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29765170-b9cc-416a-944c-393c19e88e2e admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:06.568000 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b854d4dd-2463-4745-97f7-be11df73849d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:06.569112 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b854d4dd-2463-4745-97f7-be11df73849d admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:06.569462 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b854d4dd-2463-4745-97f7-be11df73849d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:06.570385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b854d4dd-2463-4745-97f7-be11df73849d admin admin] PATH_INFO: `/v3/projects/1504d59e0dc846a9bd0904b292a02128/users/94de504324ad42fc910679ef6df09c26/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:06.584315 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:06.591836 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b854d4dd-2463-4745-97f7-be11df73849d admin admin] RBAC: Authorizing `identity:create_grant(project_id=1504d59e0dc846a9bd0904b292a02128, user_id=94de504324ad42fc910679ef6df09c26, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:06.595296 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b854d4dd-2463-4745-97f7-be11df73849d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:06.613406 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1198/2424] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:04:06 2022] PUT /identity/v3/projects/1504d59e0dc846a9bd0904b292a02128/users/94de504324ad42fc910679ef6df09c26/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:04:06.623504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6915ce-c905-49e2-b594-7e2a09876d74 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:06.624523 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6915ce-c905-49e2-b594-7e2a09876d74 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:06.625252 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6915ce-c905-49e2-b594-7e2a09876d74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:06.667775 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-2e6915ce-c905-49e2-b594-7e2a09876d74 None None] MFA Rules not processed for user `94de504324ad42fc910679ef6df09c26`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:06.669347 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2e6915ce-c905-49e2-b594-7e2a09876d74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:06.780504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1227/2425] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:06 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:06.788515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b854d4dd-2463-4745-97f7-be11df73849d admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:06.790248 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-b854d4dd-2463-4745-97f7-be11df73849d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:06.863796 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b854d4dd-2463-4745-97f7-be11df73849d admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:06.874004 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960658dc-700b-4960-a1cd-3fa969ae83f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:06.875437 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960658dc-700b-4960-a1cd-3fa969ae83f6 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:06.875834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960658dc-700b-4960-a1cd-3fa969ae83f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:06.875973 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960658dc-700b-4960-a1cd-3fa969ae83f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:06.886944 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-960658dc-700b-4960-a1cd-3fa969ae83f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:06.888581 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-960658dc-700b-4960-a1cd-3fa969ae83f6 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:06.903816 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1199/2426] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:06 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:06.906156 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6915ce-c905-49e2-b594-7e2a09876d74 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:06.907908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-2e6915ce-c905-49e2-b594-7e2a09876d74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:06.928198 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960658dc-700b-4960-a1cd-3fa969ae83f6 service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:06.986173 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:07.004141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960658dc-700b-4960-a1cd-3fa969ae83f6 service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:07.004796 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6915ce-c905-49e2-b594-7e2a09876d74 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:07.009555 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc54abc-bffe-445d-9fc5-4bd7d9f3e729 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:07.009728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a98c40-aa20-4619-911c-6881547e9204 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:07.010339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc54abc-bffe-445d-9fc5-4bd7d9f3e729 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:07.010445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a98c40-aa20-4619-911c-6881547e9204 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:07.010736 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a98c40-aa20-4619-911c-6881547e9204 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:07.010844 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc54abc-bffe-445d-9fc5-4bd7d9f3e729 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:07.011129 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc54abc-bffe-445d-9fc5-4bd7d9f3e729 admin admin] PATH_INFO: `/v3/users/94de504324ad42fc910679ef6df09c26` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:07.011225 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a98c40-aa20-4619-911c-6881547e9204 admin admin] PATH_INFO: `/v3/projects/ee9bc983690a4688903488d0968b324d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:07.012490 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc54abc-bffe-445d-9fc5-4bd7d9f3e729 admin admin] RBAC: Authorizing `identity:delete_user(user_id=94de504324ad42fc910679ef6df09c26)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:07.012605 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a98c40-aa20-4619-911c-6881547e9204 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee9bc983690a4688903488d0968b324d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:07.014509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a98c40-aa20-4619-911c-6881547e9204 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:07.014623 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc54abc-bffe-445d-9fc5-4bd7d9f3e729 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:07.076824 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2fc54abc-bffe-445d-9fc5-4bd7d9f3e729 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '94de504324ad42fc910679ef6df09c26'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.080762 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-81a98c40-aa20-4619-911c-6881547e9204 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee9bc983690a4688903488d0968b324d', 'request_id': 'req-81a98c40-aa20-4619-911c-6881547e9204'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.085569 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2fc54abc-bffe-445d-9fc5-4bd7d9f3e729 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '94de504324ad42fc910679ef6df09c26'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.086066 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2fc54abc-bffe-445d-9fc5-4bd7d9f3e729 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '94de504324ad42fc910679ef6df09c26'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.086823 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2fc54abc-bffe-445d-9fc5-4bd7d9f3e729 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '94de504324ad42fc910679ef6df09c26'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.091693 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-81a98c40-aa20-4619-911c-6881547e9204 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee9bc983690a4688903488d0968b324d', 'request_id': 'req-81a98c40-aa20-4619-911c-6881547e9204'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.099403 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-81a98c40-aa20-4619-911c-6881547e9204 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee9bc983690a4688903488d0968b324d', 'request_id': 'req-81a98c40-aa20-4619-911c-6881547e9204'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.105011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-2fc54abc-bffe-445d-9fc5-4bd7d9f3e729 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '94de504324ad42fc910679ef6df09c26'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.116566 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-81a98c40-aa20-4619-911c-6881547e9204 admin admin] The token cache is being invalidate because project ee9bc983690a4688903488d0968b324d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:04:07.117109 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-81a98c40-aa20-4619-911c-6881547e9204 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.117700 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-81a98c40-aa20-4619-911c-6881547e9204 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.119341 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1228/2427] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:04:06 2022] DELETE /identity/v3/projects/ee9bc983690a4688903488d0968b324d => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:07.121744 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1200/2428] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:04:06 2022] DELETE /identity/v3/users/94de504324ad42fc910679ef6df09c26 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:04:07.123172 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a98c40-aa20-4619-911c-6881547e9204 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:07.124984 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-81a98c40-aa20-4619-911c-6881547e9204 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:07.126095 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdf50d9-aeb4-4ea4-9034-0c30311676d9 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:07.126931 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdf50d9-aeb4-4ea4-9034-0c30311676d9 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:07.127485 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdf50d9-aeb4-4ea4-9034-0c30311676d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:07.148047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-3cdf50d9-aeb4-4ea4-9034-0c30311676d9 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:07.149305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-3cdf50d9-aeb4-4ea4-9034-0c30311676d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:07.154756 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:07.220661 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a98c40-aa20-4619-911c-6881547e9204 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:07.225962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1074a671-762d-4aa8-a2fd-600d27a3e8fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:07.226139 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1201/2429] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:07.226557 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1074a671-762d-4aa8-a2fd-600d27a3e8fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:07.226964 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1074a671-762d-4aa8-a2fd-600d27a3e8fe admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:07.227208 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1074a671-762d-4aa8-a2fd-600d27a3e8fe admin admin] PATH_INFO: `/v3/users/2a657f8f5f8d445d955eb52012cdca6e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:07.228524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1074a671-762d-4aa8-a2fd-600d27a3e8fe admin admin] RBAC: Authorizing `identity:delete_user(user_id=2a657f8f5f8d445d955eb52012cdca6e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:07.230971 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1074a671-762d-4aa8-a2fd-600d27a3e8fe admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:07.238990 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdf50d9-aeb4-4ea4-9034-0c30311676d9 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:07.241385 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-3cdf50d9-aeb4-4ea4-9034-0c30311676d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:07.296514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-1074a671-762d-4aa8-a2fd-600d27a3e8fe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2a657f8f5f8d445d955eb52012cdca6e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.297855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-1074a671-762d-4aa8-a2fd-600d27a3e8fe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2a657f8f5f8d445d955eb52012cdca6e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.312042 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-1074a671-762d-4aa8-a2fd-600d27a3e8fe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2a657f8f5f8d445d955eb52012cdca6e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.314030 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdf50d9-aeb4-4ea4-9034-0c30311676d9 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:07.320308 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae1205f-b335-4bbb-a226-1680abc47d0b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:07.322018 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-1074a671-762d-4aa8-a2fd-600d27a3e8fe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2a657f8f5f8d445d955eb52012cdca6e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.324051 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae1205f-b335-4bbb-a226-1680abc47d0b service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:07.324345 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae1205f-b335-4bbb-a226-1680abc47d0b service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:07.324652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae1205f-b335-4bbb-a226-1680abc47d0b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:07.326719 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-3ae1205f-b335-4bbb-a226-1680abc47d0b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:07.327234 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:07.333449 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-1074a671-762d-4aa8-a2fd-600d27a3e8fe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2a657f8f5f8d445d955eb52012cdca6e'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.341508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae1205f-b335-4bbb-a226-1680abc47d0b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:07.342206 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1229/2430] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:04:07 2022] DELETE /identity/v3/users/2a657f8f5f8d445d955eb52012cdca6e => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:04:07.343364 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae1205f-b335-4bbb-a226-1680abc47d0b service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:07.358119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1202/2431] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:07.725928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1074a671-762d-4aa8-a2fd-600d27a3e8fe admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:07.727416 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1074a671-762d-4aa8-a2fd-600d27a3e8fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:07.820468 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1074a671-762d-4aa8-a2fd-600d27a3e8fe admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:07.825943 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c63f31-7b95-4752-90e4-3ccb59fe7ecc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:07.826694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c63f31-7b95-4752-90e4-3ccb59fe7ecc admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:07.827217 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c63f31-7b95-4752-90e4-3ccb59fe7ecc admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:07.827341 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c63f31-7b95-4752-90e4-3ccb59fe7ecc admin admin] PATH_INFO: `/v3/projects/1504d59e0dc846a9bd0904b292a02128` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:07.828613 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c63f31-7b95-4752-90e4-3ccb59fe7ecc admin admin] RBAC: Authorizing `identity:delete_project(project_id=1504d59e0dc846a9bd0904b292a02128)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:07.830672 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c63f31-7b95-4752-90e4-3ccb59fe7ecc admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:07.839080 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae1205f-b335-4bbb-a226-1680abc47d0b service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:07.841097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-3ae1205f-b335-4bbb-a226-1680abc47d0b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:07.858354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae1205f-b335-4bbb-a226-1680abc47d0b service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:07.868392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e549c6-4d36-4ffc-87fe-fb8b64848458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:07.869053 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e549c6-4d36-4ffc-87fe-fb8b64848458 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:07.869346 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e549c6-4d36-4ffc-87fe-fb8b64848458 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:07.869643 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e549c6-4d36-4ffc-87fe-fb8b64848458 admin admin] PATH_INFO: `/v3/projects/fee24a80cbcb429db63b5573f1f5fac1` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:07.870882 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e549c6-4d36-4ffc-87fe-fb8b64848458 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fee24a80cbcb429db63b5573f1f5fac1)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:07.875797 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e549c6-4d36-4ffc-87fe-fb8b64848458 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:07.922341 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c5c63f31-7b95-4752-90e4-3ccb59fe7ecc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1504d59e0dc846a9bd0904b292a02128', 'request_id': 'req-c5c63f31-7b95-4752-90e4-3ccb59fe7ecc'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.930191 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c5c63f31-7b95-4752-90e4-3ccb59fe7ecc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1504d59e0dc846a9bd0904b292a02128', 'request_id': 'req-c5c63f31-7b95-4752-90e4-3ccb59fe7ecc'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.937949 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c5c63f31-7b95-4752-90e4-3ccb59fe7ecc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1504d59e0dc846a9bd0904b292a02128', 'request_id': 'req-c5c63f31-7b95-4752-90e4-3ccb59fe7ecc'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.939101 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a4e549c6-4d36-4ffc-87fe-fb8b64848458 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fee24a80cbcb429db63b5573f1f5fac1', 'request_id': 'req-a4e549c6-4d36-4ffc-87fe-fb8b64848458'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.954585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a4e549c6-4d36-4ffc-87fe-fb8b64848458 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fee24a80cbcb429db63b5573f1f5fac1', 'request_id': 'req-a4e549c6-4d36-4ffc-87fe-fb8b64848458'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.956419 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c5c63f31-7b95-4752-90e4-3ccb59fe7ecc admin admin] The token cache is being invalidate because project 1504d59e0dc846a9bd0904b292a02128 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:04:07.956662 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c5c63f31-7b95-4752-90e4-3ccb59fe7ecc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.958107 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-c5c63f31-7b95-4752-90e4-3ccb59fe7ecc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.960336 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1230/2432] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:04:07 2022] DELETE /identity/v3/projects/1504d59e0dc846a9bd0904b292a02128 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:07.961080 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a4e549c6-4d36-4ffc-87fe-fb8b64848458 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fee24a80cbcb429db63b5573f1f5fac1', 'request_id': 'req-a4e549c6-4d36-4ffc-87fe-fb8b64848458'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.965897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79bbe5c-a635-4405-9b56-0f4894879877 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:07.966235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79bbe5c-a635-4405-9b56-0f4894879877 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:07.966520 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79bbe5c-a635-4405-9b56-0f4894879877 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:07.973518 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a4e549c6-4d36-4ffc-87fe-fb8b64848458 admin admin] The token cache is being invalidate because project fee24a80cbcb429db63b5573f1f5fac1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:04:07.973996 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a4e549c6-4d36-4ffc-87fe-fb8b64848458 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.974654 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-a4e549c6-4d36-4ffc-87fe-fb8b64848458 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:07.976194 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1203/2433] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:04:07 2022] DELETE /identity/v3/projects/fee24a80cbcb429db63b5573f1f5fac1 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:07.980886 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3afc4ac-411b-4d90-8a5f-a937a40c322f None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:07.981175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3afc4ac-411b-4d90-8a5f-a937a40c322f None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:07.981453 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3afc4ac-411b-4d90-8a5f-a937a40c322f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:07.990601 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-b79bbe5c-a635-4405-9b56-0f4894879877 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:07.991997 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-b79bbe5c-a635-4405-9b56-0f4894879877 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:08.003997 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-d3afc4ac-411b-4d90-8a5f-a937a40c322f None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:08.005272 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d3afc4ac-411b-4d90-8a5f-a937a40c322f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:08.045093 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:08.080140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1204/2434] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:04:08.081261 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1231/2435] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:04:08.090341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51bb407-b3b5-4ede-a035-22554fa13439 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.090631 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51bb407-b3b5-4ede-a035-22554fa13439 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.090906 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51bb407-b3b5-4ede-a035-22554fa13439 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.093053 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1dafbf-2822-4e87-af29-5aa565123858 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.093347 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1dafbf-2822-4e87-af29-5aa565123858 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.093684 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1dafbf-2822-4e87-af29-5aa565123858 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.113423 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-e51bb407-b3b5-4ede-a035-22554fa13439 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:08.114884 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e51bb407-b3b5-4ede-a035-22554fa13439 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:08.121722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-0e1dafbf-2822-4e87-af29-5aa565123858 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:08.122926 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0e1dafbf-2822-4e87-af29-5aa565123858 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:08.123601 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1205/2436] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:08.127700 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51bb407-b3b5-4ede-a035-22554fa13439 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:08.134115 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1232/2437] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:08.137870 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1dafbf-2822-4e87-af29-5aa565123858 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:08.141372 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51bb407-b3b5-4ede-a035-22554fa13439 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:08.147389 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d11b6b-3d5d-42db-9d40-5c3d583f4ab9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:08.148777 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d11b6b-3d5d-42db-9d40-5c3d583f4ab9 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.149074 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d11b6b-3d5d-42db-9d40-5c3d583f4ab9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.149193 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1dafbf-2822-4e87-af29-5aa565123858 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:08.149446 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d11b6b-3d5d-42db-9d40-5c3d583f4ab9 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.150334 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d11b6b-3d5d-42db-9d40-5c3d583f4ab9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:04:08.150650 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d11b6b-3d5d-42db-9d40-5c3d583f4ab9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:08.152501 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d11b6b-3d5d-42db-9d40-5c3d583f4ab9 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:08.162904 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb02d996-74df-4361-b678-36e617e0e56a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:08.163670 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb02d996-74df-4361-b678-36e617e0e56a admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.164032 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb02d996-74df-4361-b678-36e617e0e56a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.164708 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb02d996-74df-4361-b678-36e617e0e56a admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.164966 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb02d996-74df-4361-b678-36e617e0e56a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:04:08.165275 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb02d996-74df-4361-b678-36e617e0e56a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:08.166893 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb02d996-74df-4361-b678-36e617e0e56a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:08.171067 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1206/2438] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:04:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 31 15:04:08.172158 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d11b6b-3d5d-42db-9d40-5c3d583f4ab9 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:08.182837 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1233/2439] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:04:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:04:08.183035 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d11b6b-3d5d-42db-9d40-5c3d583f4ab9 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:08.183897 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb02d996-74df-4361-b678-36e617e0e56a admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:08.188205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7792c49-d61d-4878-bbf8-0b0d8bd5db92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:08.188834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7792c49-d61d-4878-bbf8-0b0d8bd5db92 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.189112 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7792c49-d61d-4878-bbf8-0b0d8bd5db92 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.189405 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7792c49-d61d-4878-bbf8-0b0d8bd5db92 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.190031 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7792c49-d61d-4878-bbf8-0b0d8bd5db92 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:08.193523 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7792c49-d61d-4878-bbf8-0b0d8bd5db92 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:08.197715 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb02d996-74df-4361-b678-36e617e0e56a admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:08.202720 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026b5ce3-8313-42c5-aea2-7c0c31ec3883 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:08.203334 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026b5ce3-8313-42c5-aea2-7c0c31ec3883 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.203682 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026b5ce3-8313-42c5-aea2-7c0c31ec3883 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.203965 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026b5ce3-8313-42c5-aea2-7c0c31ec3883 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.204603 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026b5ce3-8313-42c5-aea2-7c0c31ec3883 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:08.206555 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026b5ce3-8313-42c5-aea2-7c0c31ec3883 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:08.212426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:08.224210 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1207/2440] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:04:08 2022] POST /identity/v3/projects => generated 369 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:04:08.228354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7792c49-d61d-4878-bbf8-0b0d8bd5db92 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:08.241609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1234/2441] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:04:08 2022] POST /identity/v3/projects => generated 383 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:04:08.245839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026b5ce3-8313-42c5-aea2-7c0c31ec3883 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:08.303116 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7792c49-d61d-4878-bbf8-0b0d8bd5db92 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:08.315928 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026b5ce3-8313-42c5-aea2-7c0c31ec3883 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:08.322713 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fa17fa-11ec-4916-80b4-77e79a69a0b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:08.323514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fa17fa-11ec-4916-80b4-77e79a69a0b6 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.323851 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fa17fa-11ec-4916-80b4-77e79a69a0b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.324164 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fa17fa-11ec-4916-80b4-77e79a69a0b6 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.325502 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fa17fa-11ec-4916-80b4-77e79a69a0b6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:08.327718 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fa17fa-11ec-4916-80b4-77e79a69a0b6 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:08.349713 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1235/2442] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:04:08 2022] POST /identity/v3/users => generated 292 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:04:08.354055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fa17fa-11ec-4916-80b4-77e79a69a0b6 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:08.364990 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fa17fa-11ec-4916-80b4-77e79a69a0b6 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:08.369908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc8619a-e10d-4b1e-8064-b972fd0f3e49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:08.370504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc8619a-e10d-4b1e-8064-b972fd0f3e49 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.370784 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc8619a-e10d-4b1e-8064-b972fd0f3e49 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.371058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc8619a-e10d-4b1e-8064-b972fd0f3e49 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.371750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc8619a-e10d-4b1e-8064-b972fd0f3e49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:08.373323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc8619a-e10d-4b1e-8064-b972fd0f3e49 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:08.375989 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6baca94d-3e03-4547-b9b7-615de5c59c19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:08.376591 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6baca94d-3e03-4547-b9b7-615de5c59c19 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.376866 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6baca94d-3e03-4547-b9b7-615de5c59c19 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.377183 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6baca94d-3e03-4547-b9b7-615de5c59c19 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.377808 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6baca94d-3e03-4547-b9b7-615de5c59c19 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:08.380682 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6baca94d-3e03-4547-b9b7-615de5c59c19 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:08.387245 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1236/2443] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:08 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:04:08.388518 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc8619a-e10d-4b1e-8064-b972fd0f3e49 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:08.399400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc8619a-e10d-4b1e-8064-b972fd0f3e49 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:08.404412 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4c483d-76e6-4d0a-9c7b-4d083e69b969 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:08.405089 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4c483d-76e6-4d0a-9c7b-4d083e69b969 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.405443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4c483d-76e6-4d0a-9c7b-4d083e69b969 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.405556 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1208/2444] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:04:08 2022] POST /identity/v3/users => generated 285 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:04:08.405719 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4c483d-76e6-4d0a-9c7b-4d083e69b969 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.406362 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4c483d-76e6-4d0a-9c7b-4d083e69b969 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:08.407978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4c483d-76e6-4d0a-9c7b-4d083e69b969 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:08.409422 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6baca94d-3e03-4547-b9b7-615de5c59c19 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:08.419036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1237/2445] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:08 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:04:08.420180 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6baca94d-3e03-4547-b9b7-615de5c59c19 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:08.423425 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4c483d-76e6-4d0a-9c7b-4d083e69b969 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:08.426879 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2374ca95-7f98-42db-87ea-8ab33145126f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:08.427495 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2374ca95-7f98-42db-87ea-8ab33145126f admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.427815 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2374ca95-7f98-42db-87ea-8ab33145126f admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.428097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2374ca95-7f98-42db-87ea-8ab33145126f admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.428746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2374ca95-7f98-42db-87ea-8ab33145126f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:08.431275 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2374ca95-7f98-42db-87ea-8ab33145126f admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:08.434010 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4c483d-76e6-4d0a-9c7b-4d083e69b969 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:08.438671 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a11b31e-288c-4d57-a622-a971a9fc9cc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:08.439317 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a11b31e-288c-4d57-a622-a971a9fc9cc4 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.439623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a11b31e-288c-4d57-a622-a971a9fc9cc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.439901 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a11b31e-288c-4d57-a622-a971a9fc9cc4 admin admin] PATH_INFO: `/v3/projects/3424a823161e49029c454a8818f7bd7f/users/7b9ebf0c707045908f3950c7077b3554/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.442937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1209/2446] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:08 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:04:08.447990 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2374ca95-7f98-42db-87ea-8ab33145126f admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:08.458354 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2374ca95-7f98-42db-87ea-8ab33145126f admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:08.461416 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a11b31e-288c-4d57-a622-a971a9fc9cc4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3424a823161e49029c454a8818f7bd7f, user_id=7b9ebf0c707045908f3950c7077b3554, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:08.463102 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c089af87-8b37-489b-a928-b9921beaa03c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:08.463743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c089af87-8b37-489b-a928-b9921beaa03c admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.464025 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c089af87-8b37-489b-a928-b9921beaa03c admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.464174 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a11b31e-288c-4d57-a622-a971a9fc9cc4 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:08.464298 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c089af87-8b37-489b-a928-b9921beaa03c admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.464922 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c089af87-8b37-489b-a928-b9921beaa03c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:08.466696 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c089af87-8b37-489b-a928-b9921beaa03c admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:08.480489 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1210/2447] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:08 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:04:08.484658 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1238/2448] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:04:08 2022] PUT /identity/v3/projects/3424a823161e49029c454a8818f7bd7f/users/7b9ebf0c707045908f3950c7077b3554/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:04:08.484658 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a11b31e-288c-4d57-a622-a971a9fc9cc4 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:08.486450 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0936a9-79e8-4468-b742-43fb6cc4d1e8 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.486737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0936a9-79e8-4468-b742-43fb6cc4d1e8 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.487016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0936a9-79e8-4468-b742-43fb6cc4d1e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.514511 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-9f0936a9-79e8-4468-b742-43fb6cc4d1e8 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:08.515754 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-9f0936a9-79e8-4468-b742-43fb6cc4d1e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:08.537185 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:08.564542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a11b31e-288c-4d57-a622-a971a9fc9cc4 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:08.574238 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ec3538-bc1e-4653-b569-9f92e79467ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:08.575289 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ec3538-bc1e-4653-b569-9f92e79467ca admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.575810 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ec3538-bc1e-4653-b569-9f92e79467ca admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.576151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ec3538-bc1e-4653-b569-9f92e79467ca admin admin] PATH_INFO: `/v3/projects/db00efbed8074c5786b311c8dd6d1534/users/d6ed58043879474883bc2b54ed6ddd0c/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.593192 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ec3538-bc1e-4653-b569-9f92e79467ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=db00efbed8074c5786b311c8dd6d1534, user_id=d6ed58043879474883bc2b54ed6ddd0c, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:08.596048 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ec3538-bc1e-4653-b569-9f92e79467ca admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:08.604804 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1211/2449] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:08.614552 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1239/2450] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:04:08 2022] PUT /identity/v3/projects/db00efbed8074c5786b311c8dd6d1534/users/d6ed58043879474883bc2b54ed6ddd0c/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:04:08.616664 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0936a9-79e8-4468-b742-43fb6cc4d1e8 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:08.618911 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-9f0936a9-79e8-4468-b742-43fb6cc4d1e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:08.625290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146b8884-c6b3-470a-aceb-66b22a160ea4 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.625854 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146b8884-c6b3-470a-aceb-66b22a160ea4 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.626230 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146b8884-c6b3-470a-aceb-66b22a160ea4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.656413 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:08.668791 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-146b8884-c6b3-470a-aceb-66b22a160ea4 None None] MFA Rules not processed for user `d6ed58043879474883bc2b54ed6ddd0c`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:08.670308 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-146b8884-c6b3-470a-aceb-66b22a160ea4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:08.703461 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0936a9-79e8-4468-b742-43fb6cc4d1e8 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:08.747664 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1240/2451] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:08 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:04:08.752199 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146b8884-c6b3-470a-aceb-66b22a160ea4 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:08.766472 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbea455-6f97-4b65-ac43-47d4feb9c4f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:08.767283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbea455-6f97-4b65-ac43-47d4feb9c4f4 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.767726 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbea455-6f97-4b65-ac43-47d4feb9c4f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.768098 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbea455-6f97-4b65-ac43-47d4feb9c4f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.823611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146b8884-c6b3-470a-aceb-66b22a160ea4 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:08.828559 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969126e2-3cb2-4686-b791-6865d22b6ebd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:08.829170 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969126e2-3cb2-4686-b791-6865d22b6ebd admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.829485 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969126e2-3cb2-4686-b791-6865d22b6ebd admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.829764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969126e2-3cb2-4686-b791-6865d22b6ebd admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.830413 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969126e2-3cb2-4686-b791-6865d22b6ebd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:08.831490 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbea455-6f97-4b65-ac43-47d4feb9c4f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:08.832408 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969126e2-3cb2-4686-b791-6865d22b6ebd admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:08.833290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbea455-6f97-4b65-ac43-47d4feb9c4f4 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:08.843026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1212/2452] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:08.867229 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1241/2453] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:04:08 2022] POST /identity/v3/projects => generated 367 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:04:08.871495 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbea455-6f97-4b65-ac43-47d4feb9c4f4 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:08.882214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbea455-6f97-4b65-ac43-47d4feb9c4f4 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:08.895821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:08.957166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bd14ec-fcbb-46da-95b4-bd3ce69bb3d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:08.957889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd14ec-fcbb-46da-95b4-bd3ce69bb3d1 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:08.958238 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd14ec-fcbb-46da-95b4-bd3ce69bb3d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:08.958508 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd14ec-fcbb-46da-95b4-bd3ce69bb3d1 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:08.959148 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bd14ec-fcbb-46da-95b4-bd3ce69bb3d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:08.961394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bd14ec-fcbb-46da-95b4-bd3ce69bb3d1 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:08.987693 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1213/2454] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:04:08 2022] POST /identity/v3/users => generated 290 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:04:08.989635 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969126e2-3cb2-4686-b791-6865d22b6ebd admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:09.001111 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969126e2-3cb2-4686-b791-6865d22b6ebd admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:09.010047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bb026b-19a9-4b2a-b800-59ee2ee206c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:09.010691 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bb026b-19a9-4b2a-b800-59ee2ee206c5 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:09.010973 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bb026b-19a9-4b2a-b800-59ee2ee206c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:09.011246 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bb026b-19a9-4b2a-b800-59ee2ee206c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:09.011932 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bb026b-19a9-4b2a-b800-59ee2ee206c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:09.013536 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bb026b-19a9-4b2a-b800-59ee2ee206c5 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:09.025334 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1242/2455] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:08 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:04:09.029529 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bd14ec-fcbb-46da-95b4-bd3ce69bb3d1 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:09.045475 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bd14ec-fcbb-46da-95b4-bd3ce69bb3d1 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:09.050663 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33713535-8256-4aa2-81a0-d58ae2ac7819 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:09.051374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33713535-8256-4aa2-81a0-d58ae2ac7819 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:09.051683 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33713535-8256-4aa2-81a0-d58ae2ac7819 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:09.051956 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33713535-8256-4aa2-81a0-d58ae2ac7819 admin admin] PATH_INFO: `/v3/projects/93d56ebf11524dc885f54d591bf2eaab/users/6af17b53de3a45c7824cd1525798c86a/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:09.069677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33713535-8256-4aa2-81a0-d58ae2ac7819 admin admin] RBAC: Authorizing `identity:create_grant(project_id=93d56ebf11524dc885f54d591bf2eaab, user_id=6af17b53de3a45c7824cd1525798c86a, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:09.072557 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33713535-8256-4aa2-81a0-d58ae2ac7819 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:09.090977 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1214/2456] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:04:09 2022] PUT /identity/v3/projects/93d56ebf11524dc885f54d591bf2eaab/users/6af17b53de3a45c7824cd1525798c86a/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:09.092208 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bb026b-19a9-4b2a-b800-59ee2ee206c5 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:09.105673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bb026b-19a9-4b2a-b800-59ee2ee206c5 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:09.171101 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b04b27e-472b-47fd-a2a1-f0c028c83d3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:09.171765 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b04b27e-472b-47fd-a2a1-f0c028c83d3a admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:09.172051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b04b27e-472b-47fd-a2a1-f0c028c83d3a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:09.172328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b04b27e-472b-47fd-a2a1-f0c028c83d3a admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:09.172981 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b04b27e-472b-47fd-a2a1-f0c028c83d3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:09.175092 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b04b27e-472b-47fd-a2a1-f0c028c83d3a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:09.186758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1243/2457] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:09 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:04:09.191392 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33713535-8256-4aa2-81a0-d58ae2ac7819 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:09.204525 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33713535-8256-4aa2-81a0-d58ae2ac7819 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:09.210813 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44113cdc-2c4b-483c-a8f3-515b35a1cb1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:09.212943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44113cdc-2c4b-483c-a8f3-515b35a1cb1d admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:09.213336 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44113cdc-2c4b-483c-a8f3-515b35a1cb1d admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:09.213737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44113cdc-2c4b-483c-a8f3-515b35a1cb1d admin admin] PATH_INFO: `/v3/projects/93d56ebf11524dc885f54d591bf2eaab/users/6af17b53de3a45c7824cd1525798c86a/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:09.216310 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44113cdc-2c4b-483c-a8f3-515b35a1cb1d admin admin] RBAC: Authorizing `identity:create_grant(project_id=93d56ebf11524dc885f54d591bf2eaab, user_id=6af17b53de3a45c7824cd1525798c86a, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:09.219127 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44113cdc-2c4b-483c-a8f3-515b35a1cb1d admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:09.235237 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1215/2458] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:04:09 2022] PUT /identity/v3/projects/93d56ebf11524dc885f54d591bf2eaab/users/6af17b53de3a45c7824cd1525798c86a/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:09.246788 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5768209-aef0-44c5-ba1f-90a786f302c3 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:09.247070 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5768209-aef0-44c5-ba1f-90a786f302c3 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:09.247343 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5768209-aef0-44c5-ba1f-90a786f302c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:09.271955 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:09.282332 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-c5768209-aef0-44c5-ba1f-90a786f302c3 None None] MFA Rules not processed for user `6af17b53de3a45c7824cd1525798c86a`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:09.283767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-c5768209-aef0-44c5-ba1f-90a786f302c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:09.366354 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1244/2459] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:09 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:09.377267 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44113cdc-2c4b-483c-a8f3-515b35a1cb1d admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:09.379087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-44113cdc-2c4b-483c-a8f3-515b35a1cb1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:09.445496 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44113cdc-2c4b-483c-a8f3-515b35a1cb1d admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:09.449273 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ccb067-400b-4540-a8b3-6ac974167421 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:09.449885 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ccb067-400b-4540-a8b3-6ac974167421 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:09.450167 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ccb067-400b-4540-a8b3-6ac974167421 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:09.450439 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ccb067-400b-4540-a8b3-6ac974167421 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:09.465021 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ccb067-400b-4540-a8b3-6ac974167421 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:09.466527 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ccb067-400b-4540-a8b3-6ac974167421 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:09.477331 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1216/2460] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:09 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:09.529976 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed5236a-e0bd-43ec-b36e-ac2cd3c1fad3 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:09.530372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed5236a-e0bd-43ec-b36e-ac2cd3c1fad3 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:09.530657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed5236a-e0bd-43ec-b36e-ac2cd3c1fad3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:09.537669 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ccb067-400b-4540-a8b3-6ac974167421 service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:09.537669 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-f0ccb067-400b-4540-a8b3-6ac974167421 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:09.549334 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ccb067-400b-4540-a8b3-6ac974167421 service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:09.556574 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19a83a5-9d4e-44b9-8e89-aa150c2bc873 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:09.558037 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19a83a5-9d4e-44b9-8e89-aa150c2bc873 service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:09.558331 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19a83a5-9d4e-44b9-8e89-aa150c2bc873 service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:09.558609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19a83a5-9d4e-44b9-8e89-aa150c2bc873 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:09.565054 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-7ed5236a-e0bd-43ec-b36e-ac2cd3c1fad3 None None] MFA Rules not processed for user `7b9ebf0c707045908f3950c7077b3554`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:09.567352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-7ed5236a-e0bd-43ec-b36e-ac2cd3c1fad3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:09.646422 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19a83a5-9d4e-44b9-8e89-aa150c2bc873 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:09.647937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19a83a5-9d4e-44b9-8e89-aa150c2bc873 service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:09.653646 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1245/2461] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:09 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:09.663119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1217/2462] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:09 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 15:04:09.663834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed5236a-e0bd-43ec-b36e-ac2cd3c1fad3 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:09.675121 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed5236a-e0bd-43ec-b36e-ac2cd3c1fad3 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:09.684182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cc0355-ba51-44e7-90a7-ed10d9fc7c0d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:09.684812 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cc0355-ba51-44e7-90a7-ed10d9fc7c0d service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:09.685091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cc0355-ba51-44e7-90a7-ed10d9fc7c0d service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:09.685364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cc0355-ba51-44e7-90a7-ed10d9fc7c0d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:09.695323 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cc0355-ba51-44e7-90a7-ed10d9fc7c0d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:09.696822 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cc0355-ba51-44e7-90a7-ed10d9fc7c0d service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:09.706008 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1246/2463] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:09 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 15:04:11.159457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434ab6d2-c087-4cd9-a82a-116642e54536 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:11.160486 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434ab6d2-c087-4cd9-a82a-116642e54536 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:11.160714 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434ab6d2-c087-4cd9-a82a-116642e54536 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:11.181720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-434ab6d2-c087-4cd9-a82a-116642e54536 None None] MFA Rules not processed for user `7b9ebf0c707045908f3950c7077b3554`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:11.182959 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-434ab6d2-c087-4cd9-a82a-116642e54536 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:11.189974 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1218/2464] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:11 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:13.101145 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cc0355-ba51-44e7-90a7-ed10d9fc7c0d service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:13.165350 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cc0355-ba51-44e7-90a7-ed10d9fc7c0d service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:13.170055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a729a0-a0ca-4b07-b2e6-84afe2f0589d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:13.170674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a729a0-a0ca-4b07-b2e6-84afe2f0589d admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:13.170979 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a729a0-a0ca-4b07-b2e6-84afe2f0589d admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:13.171256 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a729a0-a0ca-4b07-b2e6-84afe2f0589d admin admin] PATH_INFO: `/v3/users/d6ed58043879474883bc2b54ed6ddd0c` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:13.172408 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a729a0-a0ca-4b07-b2e6-84afe2f0589d admin admin] RBAC: Authorizing `identity:delete_user(user_id=d6ed58043879474883bc2b54ed6ddd0c)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:13.174309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a729a0-a0ca-4b07-b2e6-84afe2f0589d admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:13.188056 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:13.223944 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-63a729a0-a0ca-4b07-b2e6-84afe2f0589d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd6ed58043879474883bc2b54ed6ddd0c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:13.224593 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-63a729a0-a0ca-4b07-b2e6-84afe2f0589d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd6ed58043879474883bc2b54ed6ddd0c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:13.236201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-63a729a0-a0ca-4b07-b2e6-84afe2f0589d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd6ed58043879474883bc2b54ed6ddd0c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:13.242961 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-63a729a0-a0ca-4b07-b2e6-84afe2f0589d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd6ed58043879474883bc2b54ed6ddd0c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:13.250225 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-63a729a0-a0ca-4b07-b2e6-84afe2f0589d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd6ed58043879474883bc2b54ed6ddd0c'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:13.257069 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1247/2465] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:04:13 2022] DELETE /identity/v3/users/d6ed58043879474883bc2b54ed6ddd0c => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:13.261668 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331e2022-691e-4b42-9363-0ac737c156b4 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:13.262137 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331e2022-691e-4b42-9363-0ac737c156b4 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:13.262307 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331e2022-691e-4b42-9363-0ac737c156b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:13.270990 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:13.282757 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-331e2022-691e-4b42-9363-0ac737c156b4 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:13.284130 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-331e2022-691e-4b42-9363-0ac737c156b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:13.358226 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1219/2466] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:13.368509 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a729a0-a0ca-4b07-b2e6-84afe2f0589d admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:13.369983 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-63a729a0-a0ca-4b07-b2e6-84afe2f0589d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:13.444144 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a729a0-a0ca-4b07-b2e6-84afe2f0589d admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:13.448117 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565a5254-ee74-4239-8545-74f6e93330c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:13.448965 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565a5254-ee74-4239-8545-74f6e93330c3 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:13.449399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565a5254-ee74-4239-8545-74f6e93330c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:13.449959 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565a5254-ee74-4239-8545-74f6e93330c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:13.460850 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565a5254-ee74-4239-8545-74f6e93330c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:13.462581 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565a5254-ee74-4239-8545-74f6e93330c3 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:13.473886 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1248/2467] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:13.798681 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331e2022-691e-4b42-9363-0ac737c156b4 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:13.800681 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-331e2022-691e-4b42-9363-0ac737c156b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:13.819233 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331e2022-691e-4b42-9363-0ac737c156b4 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:13.831436 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d511660c-50f8-4fc2-b6d3-a4434d5d3594 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:13.832122 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d511660c-50f8-4fc2-b6d3-a4434d5d3594 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:13.832404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d511660c-50f8-4fc2-b6d3-a4434d5d3594 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:13.832678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d511660c-50f8-4fc2-b6d3-a4434d5d3594 admin admin] PATH_INFO: `/v3/projects/db00efbed8074c5786b311c8dd6d1534` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:13.833817 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d511660c-50f8-4fc2-b6d3-a4434d5d3594 admin admin] RBAC: Authorizing `identity:delete_project(project_id=db00efbed8074c5786b311c8dd6d1534)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:13.835778 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d511660c-50f8-4fc2-b6d3-a4434d5d3594 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:13.903736 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d511660c-50f8-4fc2-b6d3-a4434d5d3594 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db00efbed8074c5786b311c8dd6d1534', 'request_id': 'req-d511660c-50f8-4fc2-b6d3-a4434d5d3594'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:13.914689 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d511660c-50f8-4fc2-b6d3-a4434d5d3594 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db00efbed8074c5786b311c8dd6d1534', 'request_id': 'req-d511660c-50f8-4fc2-b6d3-a4434d5d3594'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:13.921039 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d511660c-50f8-4fc2-b6d3-a4434d5d3594 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db00efbed8074c5786b311c8dd6d1534', 'request_id': 'req-d511660c-50f8-4fc2-b6d3-a4434d5d3594'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:13.933364 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d511660c-50f8-4fc2-b6d3-a4434d5d3594 admin admin] The token cache is being invalidate because project db00efbed8074c5786b311c8dd6d1534 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:04:13.933737 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d511660c-50f8-4fc2-b6d3-a4434d5d3594 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:13.934404 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d511660c-50f8-4fc2-b6d3-a4434d5d3594 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:13.935957 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1220/2468] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:04:13 2022] DELETE /identity/v3/projects/db00efbed8074c5786b311c8dd6d1534 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:13.940050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565a5254-ee74-4239-8545-74f6e93330c3 service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:13.951493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565a5254-ee74-4239-8545-74f6e93330c3 service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:14.016737 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc9f32b-f49c-42ba-81ca-3d2ffb777cee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:14.017307 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc9f32b-f49c-42ba-81ca-3d2ffb777cee admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:14.017586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc9f32b-f49c-42ba-81ca-3d2ffb777cee admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:14.017860 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc9f32b-f49c-42ba-81ca-3d2ffb777cee admin admin] PATH_INFO: `/v3/users/6af17b53de3a45c7824cd1525798c86a` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:14.019390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc9f32b-f49c-42ba-81ca-3d2ffb777cee admin admin] RBAC: Authorizing `identity:delete_user(user_id=6af17b53de3a45c7824cd1525798c86a)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:14.021568 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc9f32b-f49c-42ba-81ca-3d2ffb777cee admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:14.064395 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:14.075236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0bc9f32b-f49c-42ba-81ca-3d2ffb777cee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6af17b53de3a45c7824cd1525798c86a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:14.076163 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0bc9f32b-f49c-42ba-81ca-3d2ffb777cee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6af17b53de3a45c7824cd1525798c86a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:14.089876 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0bc9f32b-f49c-42ba-81ca-3d2ffb777cee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6af17b53de3a45c7824cd1525798c86a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:14.097976 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0bc9f32b-f49c-42ba-81ca-3d2ffb777cee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6af17b53de3a45c7824cd1525798c86a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:14.105390 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0bc9f32b-f49c-42ba-81ca-3d2ffb777cee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6af17b53de3a45c7824cd1525798c86a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:14.112225 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1249/2469] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:04:13 2022] DELETE /identity/v3/users/6af17b53de3a45c7824cd1525798c86a => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:04:14.535494 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d511660c-50f8-4fc2-b6d3-a4434d5d3594 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:14.537069 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d511660c-50f8-4fc2-b6d3-a4434d5d3594 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:14.550905 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d511660c-50f8-4fc2-b6d3-a4434d5d3594 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:14.627919 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57cd4f2-f3ef-4e1c-8b55-52e7765ac371 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:14.628605 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57cd4f2-f3ef-4e1c-8b55-52e7765ac371 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:14.628890 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57cd4f2-f3ef-4e1c-8b55-52e7765ac371 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:14.629165 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57cd4f2-f3ef-4e1c-8b55-52e7765ac371 admin admin] PATH_INFO: `/v3/projects/93d56ebf11524dc885f54d591bf2eaab` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:14.630322 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57cd4f2-f3ef-4e1c-8b55-52e7765ac371 admin admin] RBAC: Authorizing `identity:delete_project(project_id=93d56ebf11524dc885f54d591bf2eaab)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:14.632563 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57cd4f2-f3ef-4e1c-8b55-52e7765ac371 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:14.671640 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:14.688987 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f57cd4f2-f3ef-4e1c-8b55-52e7765ac371 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '93d56ebf11524dc885f54d591bf2eaab', 'request_id': 'req-f57cd4f2-f3ef-4e1c-8b55-52e7765ac371'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:14.697821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f57cd4f2-f3ef-4e1c-8b55-52e7765ac371 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '93d56ebf11524dc885f54d591bf2eaab', 'request_id': 'req-f57cd4f2-f3ef-4e1c-8b55-52e7765ac371'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:14.704698 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f57cd4f2-f3ef-4e1c-8b55-52e7765ac371 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '93d56ebf11524dc885f54d591bf2eaab', 'request_id': 'req-f57cd4f2-f3ef-4e1c-8b55-52e7765ac371'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:14.723530 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f57cd4f2-f3ef-4e1c-8b55-52e7765ac371 admin admin] The token cache is being invalidate because project 93d56ebf11524dc885f54d591bf2eaab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:04:14.723823 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f57cd4f2-f3ef-4e1c-8b55-52e7765ac371 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:14.724441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-f57cd4f2-f3ef-4e1c-8b55-52e7765ac371 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:14.725995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1221/2470] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:04:14 2022] DELETE /identity/v3/projects/93d56ebf11524dc885f54d591bf2eaab => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:14.730546 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9be53ce-4f66-421f-b74f-834913110f60 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:14.730839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9be53ce-4f66-421f-b74f-834913110f60 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:14.731116 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9be53ce-4f66-421f-b74f-834913110f60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:14.751659 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-f9be53ce-4f66-421f-b74f-834913110f60 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:14.752871 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-f9be53ce-4f66-421f-b74f-834913110f60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:14.814065 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1250/2471] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:14.823963 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4191e6-5d1e-46bf-bafd-a4a32c72bdec None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:14.824246 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4191e6-5d1e-46bf-bafd-a4a32c72bdec None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:14.824519 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4191e6-5d1e-46bf-bafd-a4a32c72bdec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:14.845593 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-ea4191e6-5d1e-46bf-bafd-a4a32c72bdec None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:14.846795 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ea4191e6-5d1e-46bf-bafd-a4a32c72bdec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:14.856222 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1222/2472] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:14.860528 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9be53ce-4f66-421f-b74f-834913110f60 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:14.870803 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9be53ce-4f66-421f-b74f-834913110f60 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:14.875443 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663cde15-0d17-43d7-8b3d-9a7b23fad4c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:14.876085 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663cde15-0d17-43d7-8b3d-9a7b23fad4c6 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:14.876364 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663cde15-0d17-43d7-8b3d-9a7b23fad4c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:14.876637 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663cde15-0d17-43d7-8b3d-9a7b23fad4c6 admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:14.877282 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663cde15-0d17-43d7-8b3d-9a7b23fad4c6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:04:14.877586 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663cde15-0d17-43d7-8b3d-9a7b23fad4c6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:14.879946 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663cde15-0d17-43d7-8b3d-9a7b23fad4c6 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:14.892052 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1251/2473] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:04:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:04:14.897381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4191e6-5d1e-46bf-bafd-a4a32c72bdec None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:14.906679 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4191e6-5d1e-46bf-bafd-a4a32c72bdec None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:14.917785 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9589f808-8438-4860-ba4b-2e4e22eb7db5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:14.918325 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9589f808-8438-4860-ba4b-2e4e22eb7db5 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:14.918701 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9589f808-8438-4860-ba4b-2e4e22eb7db5 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:14.919096 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9589f808-8438-4860-ba4b-2e4e22eb7db5 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:14.919876 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9589f808-8438-4860-ba4b-2e4e22eb7db5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:14.922557 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9589f808-8438-4860-ba4b-2e4e22eb7db5 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:14.973205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1223/2474] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:04:14 2022] POST /identity/v3/projects => generated 387 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:04:14.977638 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663cde15-0d17-43d7-8b3d-9a7b23fad4c6 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:14.988525 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663cde15-0d17-43d7-8b3d-9a7b23fad4c6 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:15.053205 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1468e7cb-6456-45a6-9106-972eea6d3735 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:15.053823 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1468e7cb-6456-45a6-9106-972eea6d3735 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:15.054108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1468e7cb-6456-45a6-9106-972eea6d3735 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:15.054389 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1468e7cb-6456-45a6-9106-972eea6d3735 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:15.055199 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1468e7cb-6456-45a6-9106-972eea6d3735 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:15.058177 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1468e7cb-6456-45a6-9106-972eea6d3735 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:15.086780 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1252/2475] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:04:14 2022] POST /identity/v3/users => generated 294 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:04:15.087921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9589f808-8438-4860-ba4b-2e4e22eb7db5 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:15.098577 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9589f808-8438-4860-ba4b-2e4e22eb7db5 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:15.103361 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79075178-7f21-4234-ae45-aeb7b632a3a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:15.103984 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79075178-7f21-4234-ae45-aeb7b632a3a9 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:15.104260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79075178-7f21-4234-ae45-aeb7b632a3a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:15.104531 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79075178-7f21-4234-ae45-aeb7b632a3a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:15.105170 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79075178-7f21-4234-ae45-aeb7b632a3a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:15.106760 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79075178-7f21-4234-ae45-aeb7b632a3a9 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:15.117598 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1224/2476] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:04:15.121690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1468e7cb-6456-45a6-9106-972eea6d3735 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:15.132999 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1468e7cb-6456-45a6-9106-972eea6d3735 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:15.137751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c8099f-7cd9-4c82-b5c1-ce834bee9567 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:15.138351 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c8099f-7cd9-4c82-b5c1-ce834bee9567 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:15.138631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c8099f-7cd9-4c82-b5c1-ce834bee9567 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:15.138905 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c8099f-7cd9-4c82-b5c1-ce834bee9567 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:15.139563 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c8099f-7cd9-4c82-b5c1-ce834bee9567 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:15.141162 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c8099f-7cd9-4c82-b5c1-ce834bee9567 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:15.155000 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1253/2477] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:15 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:04:15.156476 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79075178-7f21-4234-ae45-aeb7b632a3a9 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:15.167072 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79075178-7f21-4234-ae45-aeb7b632a3a9 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:15.171871 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f2e70c-aff4-4294-a14c-aa53e9fc3275 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:15.172522 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f2e70c-aff4-4294-a14c-aa53e9fc3275 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:15.172800 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f2e70c-aff4-4294-a14c-aa53e9fc3275 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:15.173076 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f2e70c-aff4-4294-a14c-aa53e9fc3275 admin admin] PATH_INFO: `/v3/projects/9d93667365554241b2ecba8d4978493b/users/2d4dfafa211c4b898c24f2c9d3bd8077/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:15.188479 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f2e70c-aff4-4294-a14c-aa53e9fc3275 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d93667365554241b2ecba8d4978493b, user_id=2d4dfafa211c4b898c24f2c9d3bd8077, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:15.191798 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f2e70c-aff4-4294-a14c-aa53e9fc3275 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:15.205159 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1225/2478] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:04:15 2022] PUT /identity/v3/projects/9d93667365554241b2ecba8d4978493b/users/2d4dfafa211c4b898c24f2c9d3bd8077/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:15.215528 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac5a129-06ba-48d6-8ddf-08e821fc804b None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:15.215852 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac5a129-06ba-48d6-8ddf-08e821fc804b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:15.216140 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac5a129-06ba-48d6-8ddf-08e821fc804b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:15.248758 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-bac5a129-06ba-48d6-8ddf-08e821fc804b None None] MFA Rules not processed for user `2d4dfafa211c4b898c24f2c9d3bd8077`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:15.250054 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-bac5a129-06ba-48d6-8ddf-08e821fc804b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:15.261552 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:15.350452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1254/2479] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:15 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:15.354831 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f2e70c-aff4-4294-a14c-aa53e9fc3275 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:15.428285 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f2e70c-aff4-4294-a14c-aa53e9fc3275 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:15.433462 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5aedb2-730d-4721-b0b9-e3b0e671f473 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:15.434092 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5aedb2-730d-4721-b0b9-e3b0e671f473 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:15.434380 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5aedb2-730d-4721-b0b9-e3b0e671f473 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:15.434652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5aedb2-730d-4721-b0b9-e3b0e671f473 admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:15.435300 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5aedb2-730d-4721-b0b9-e3b0e671f473 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:15.437276 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5aedb2-730d-4721-b0b9-e3b0e671f473 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:15.472667 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac5a129-06ba-48d6-8ddf-08e821fc804b None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:15.478145 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1226/2480] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:04:15 2022] POST /identity/v3/projects => generated 389 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:04:15.483821 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac5a129-06ba-48d6-8ddf-08e821fc804b None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:15.546864 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df85ff42-1d48-493d-8916-467d9984d0fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:15.547950 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df85ff42-1d48-493d-8916-467d9984d0fe admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:15.547950 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df85ff42-1d48-493d-8916-467d9984d0fe admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:15.548171 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df85ff42-1d48-493d-8916-467d9984d0fe admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:15.548783 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df85ff42-1d48-493d-8916-467d9984d0fe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:15.550777 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df85ff42-1d48-493d-8916-467d9984d0fe admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:15.576337 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1255/2481] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:04:15 2022] POST /identity/v3/users => generated 301 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:04:15.578041 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5aedb2-730d-4721-b0b9-e3b0e671f473 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:15.591574 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5aedb2-730d-4721-b0b9-e3b0e671f473 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:15.596361 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f9d64f-1e18-4792-8f24-4c4c8765bdf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:15.596962 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f9d64f-1e18-4792-8f24-4c4c8765bdf7 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:15.597239 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f9d64f-1e18-4792-8f24-4c4c8765bdf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:15.597513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f9d64f-1e18-4792-8f24-4c4c8765bdf7 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:15.598199 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f9d64f-1e18-4792-8f24-4c4c8765bdf7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:15.599754 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f9d64f-1e18-4792-8f24-4c4c8765bdf7 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:15.610424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1227/2482] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:15 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:04:15.614720 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df85ff42-1d48-493d-8916-467d9984d0fe admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:15.625011 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df85ff42-1d48-493d-8916-467d9984d0fe admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:15.630657 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df527d63-9d42-4bc1-b3a2-7ddc9a9d4fa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:15.631322 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df527d63-9d42-4bc1-b3a2-7ddc9a9d4fa1 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:15.631636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df527d63-9d42-4bc1-b3a2-7ddc9a9d4fa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:15.631918 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df527d63-9d42-4bc1-b3a2-7ddc9a9d4fa1 admin admin] PATH_INFO: `/v3/projects/40a705cb20e140c49c92ec26fd4e95a7/users/ceb491d566c3419ca8c87565366416ce/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:15.649576 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df527d63-9d42-4bc1-b3a2-7ddc9a9d4fa1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=40a705cb20e140c49c92ec26fd4e95a7, user_id=ceb491d566c3419ca8c87565366416ce, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:15.652266 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df527d63-9d42-4bc1-b3a2-7ddc9a9d4fa1 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:15.666363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1256/2483] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:04:15 2022] PUT /identity/v3/projects/40a705cb20e140c49c92ec26fd4e95a7/users/ceb491d566c3419ca8c87565366416ce/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:04:15.670363 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f9d64f-1e18-4792-8f24-4c4c8765bdf7 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:15.680652 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f9d64f-1e18-4792-8f24-4c4c8765bdf7 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:15.709340 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:15.743535 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d22505-c623-4c18-9166-801dcdfc4c52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:15.744173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d22505-c623-4c18-9166-801dcdfc4c52 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:15.744455 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d22505-c623-4c18-9166-801dcdfc4c52 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:15.744753 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d22505-c623-4c18-9166-801dcdfc4c52 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:15.745396 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d22505-c623-4c18-9166-801dcdfc4c52 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:15.746978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d22505-c623-4c18-9166-801dcdfc4c52 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:15.758184 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1228/2484] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:15 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 31 15:04:15.762216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df527d63-9d42-4bc1-b3a2-7ddc9a9d4fa1 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:15.773277 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df527d63-9d42-4bc1-b3a2-7ddc9a9d4fa1 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:15.778126 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42122719-e6b1-47f5-bbaa-2e06ad56072a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:15.778772 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42122719-e6b1-47f5-bbaa-2e06ad56072a admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:15.779047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42122719-e6b1-47f5-bbaa-2e06ad56072a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:15.779313 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42122719-e6b1-47f5-bbaa-2e06ad56072a admin admin] PATH_INFO: `/v3/projects/40a705cb20e140c49c92ec26fd4e95a7/users/ceb491d566c3419ca8c87565366416ce/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:15.781379 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42122719-e6b1-47f5-bbaa-2e06ad56072a admin admin] RBAC: Authorizing `identity:create_grant(project_id=40a705cb20e140c49c92ec26fd4e95a7, user_id=ceb491d566c3419ca8c87565366416ce, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:15.784034 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42122719-e6b1-47f5-bbaa-2e06ad56072a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:15.794760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1257/2485] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:04:15 2022] PUT /identity/v3/projects/40a705cb20e140c49c92ec26fd4e95a7/users/ceb491d566c3419ca8c87565366416ce/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:15.806936 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb49c6e2-89a2-4200-bf04-96ece00014f7 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:15.807248 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb49c6e2-89a2-4200-bf04-96ece00014f7 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:15.807522 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb49c6e2-89a2-4200-bf04-96ece00014f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:15.837605 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-cb49c6e2-89a2-4200-bf04-96ece00014f7 None None] MFA Rules not processed for user `ceb491d566c3419ca8c87565366416ce`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:15.838893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-cb49c6e2-89a2-4200-bf04-96ece00014f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:15.909966 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1229/2486] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:15 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:15.920953 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42122719-e6b1-47f5-bbaa-2e06ad56072a admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:15.922741 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-42122719-e6b1-47f5-bbaa-2e06ad56072a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:15.981575 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42122719-e6b1-47f5-bbaa-2e06ad56072a admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:15.985480 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6fa23a-95c8-4f25-99f5-2ead2a8779a8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:15.986199 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6fa23a-95c8-4f25-99f5-2ead2a8779a8 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:15.986483 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6fa23a-95c8-4f25-99f5-2ead2a8779a8 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:15.986757 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6fa23a-95c8-4f25-99f5-2ead2a8779a8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:15.997609 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6fa23a-95c8-4f25-99f5-2ead2a8779a8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:15.999331 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6fa23a-95c8-4f25-99f5-2ead2a8779a8 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:16.010968 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1258/2487] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:15 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:16.099387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90aec002-85e4-4d73-97f1-ce8bba8242da None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:16.099763 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90aec002-85e4-4d73-97f1-ce8bba8242da None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:16.100075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90aec002-85e4-4d73-97f1-ce8bba8242da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:16.131048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-90aec002-85e4-4d73-97f1-ce8bba8242da None None] MFA Rules not processed for user `f0b358738b5a46d8bc1db1da68040c8f`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:16.132414 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-90aec002-85e4-4d73-97f1-ce8bba8242da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:16.169056 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6fa23a-95c8-4f25-99f5-2ead2a8779a8 service cinder] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:16.176303 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:16.200828 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1230/2488] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:16 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:16.242486 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6fa23a-95c8-4f25-99f5-2ead2a8779a8 service cinder] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:16.247277 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3631aa3b-c899-41cb-8417-2c91872f0dc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:16.248078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3631aa3b-c899-41cb-8417-2c91872f0dc6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:16.248439 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3631aa3b-c899-41cb-8417-2c91872f0dc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:16.248790 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3631aa3b-c899-41cb-8417-2c91872f0dc6 admin admin] PATH_INFO: `/v3/users/2d4dfafa211c4b898c24f2c9d3bd8077` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:16.250026 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3631aa3b-c899-41cb-8417-2c91872f0dc6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d4dfafa211c4b898c24f2c9d3bd8077)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:16.252060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3631aa3b-c899-41cb-8417-2c91872f0dc6 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:16.302156 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-3631aa3b-c899-41cb-8417-2c91872f0dc6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d4dfafa211c4b898c24f2c9d3bd8077'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:16.302810 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-3631aa3b-c899-41cb-8417-2c91872f0dc6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d4dfafa211c4b898c24f2c9d3bd8077'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:16.315244 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-3631aa3b-c899-41cb-8417-2c91872f0dc6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d4dfafa211c4b898c24f2c9d3bd8077'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:16.323381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-3631aa3b-c899-41cb-8417-2c91872f0dc6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d4dfafa211c4b898c24f2c9d3bd8077'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:16.334927 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-3631aa3b-c899-41cb-8417-2c91872f0dc6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d4dfafa211c4b898c24f2c9d3bd8077'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:16.341886 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1259/2489] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:04:16 2022] DELETE /identity/v3/users/2d4dfafa211c4b898c24f2c9d3bd8077 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:04:16.346152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e18daf0-1cd4-49e4-baad-284abf3ae766 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:16.346440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e18daf0-1cd4-49e4-baad-284abf3ae766 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:16.346815 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e18daf0-1cd4-49e4-baad-284abf3ae766 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:16.367541 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-6e18daf0-1cd4-49e4-baad-284abf3ae766 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:16.368969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6e18daf0-1cd4-49e4-baad-284abf3ae766 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:16.428479 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:16.445931 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1231/2490] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:16.458026 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3631aa3b-c899-41cb-8417-2c91872f0dc6 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:16.459498 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-3631aa3b-c899-41cb-8417-2c91872f0dc6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:16.546208 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3631aa3b-c899-41cb-8417-2c91872f0dc6 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:16.552913 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396b6ab2-c446-458e-b7a9-a0e2e135e375 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:16.554006 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396b6ab2-c446-458e-b7a9-a0e2e135e375 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:16.555043 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396b6ab2-c446-458e-b7a9-a0e2e135e375 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:16.555345 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396b6ab2-c446-458e-b7a9-a0e2e135e375 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:16.571108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396b6ab2-c446-458e-b7a9-a0e2e135e375 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:16.572601 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396b6ab2-c446-458e-b7a9-a0e2e135e375 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:16.595824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1260/2491] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 31 15:04:17.085249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e18daf0-1cd4-49e4-baad-284abf3ae766 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:17.089004 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-6e18daf0-1cd4-49e4-baad-284abf3ae766 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:17.122412 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e18daf0-1cd4-49e4-baad-284abf3ae766 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:17.129017 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59e7f39-a15b-4709-b75d-8a28dba3de14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:17.129762 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59e7f39-a15b-4709-b75d-8a28dba3de14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:17.130446 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59e7f39-a15b-4709-b75d-8a28dba3de14 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:17.131039 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59e7f39-a15b-4709-b75d-8a28dba3de14 admin admin] PATH_INFO: `/v3/projects/9d93667365554241b2ecba8d4978493b` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:17.132290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59e7f39-a15b-4709-b75d-8a28dba3de14 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9d93667365554241b2ecba8d4978493b)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:17.135011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59e7f39-a15b-4709-b75d-8a28dba3de14 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:17.186111 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b59e7f39-a15b-4709-b75d-8a28dba3de14 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9d93667365554241b2ecba8d4978493b', 'request_id': 'req-b59e7f39-a15b-4709-b75d-8a28dba3de14'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:17.194026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b59e7f39-a15b-4709-b75d-8a28dba3de14 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9d93667365554241b2ecba8d4978493b', 'request_id': 'req-b59e7f39-a15b-4709-b75d-8a28dba3de14'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:17.200019 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b59e7f39-a15b-4709-b75d-8a28dba3de14 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9d93667365554241b2ecba8d4978493b', 'request_id': 'req-b59e7f39-a15b-4709-b75d-8a28dba3de14'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:17.211745 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b59e7f39-a15b-4709-b75d-8a28dba3de14 admin admin] The token cache is being invalidate because project 9d93667365554241b2ecba8d4978493b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:04:17.212251 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b59e7f39-a15b-4709-b75d-8a28dba3de14 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:17.213426 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-b59e7f39-a15b-4709-b75d-8a28dba3de14 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:17.217374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1232/2492] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:04:17 2022] DELETE /identity/v3/projects/9d93667365554241b2ecba8d4978493b => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:17.218516 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396b6ab2-c446-458e-b7a9-a0e2e135e375 service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:17.229595 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396b6ab2-c446-458e-b7a9-a0e2e135e375 service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:17.259046 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:17.303908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4191afc3-f3ba-4ae2-974d-253eb7003a58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:17.304535 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4191afc3-f3ba-4ae2-974d-253eb7003a58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:17.304815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4191afc3-f3ba-4ae2-974d-253eb7003a58 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:17.305093 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4191afc3-f3ba-4ae2-974d-253eb7003a58 admin admin] PATH_INFO: `/v3/users/ceb491d566c3419ca8c87565366416ce` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:17.306206 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4191afc3-f3ba-4ae2-974d-253eb7003a58 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ceb491d566c3419ca8c87565366416ce)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:17.308122 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4191afc3-f3ba-4ae2-974d-253eb7003a58 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:17.360664 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-4191afc3-f3ba-4ae2-974d-253eb7003a58 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ceb491d566c3419ca8c87565366416ce'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:17.361398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-4191afc3-f3ba-4ae2-974d-253eb7003a58 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ceb491d566c3419ca8c87565366416ce'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:17.373759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-4191afc3-f3ba-4ae2-974d-253eb7003a58 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ceb491d566c3419ca8c87565366416ce'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:17.380552 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-4191afc3-f3ba-4ae2-974d-253eb7003a58 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ceb491d566c3419ca8c87565366416ce'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:17.388523 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-4191afc3-f3ba-4ae2-974d-253eb7003a58 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ceb491d566c3419ca8c87565366416ce'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:17.394913 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1261/2493] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:04:17 2022] DELETE /identity/v3/users/ceb491d566c3419ca8c87565366416ce => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:17.680558 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59e7f39-a15b-4709-b75d-8a28dba3de14 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:17.682142 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-b59e7f39-a15b-4709-b75d-8a28dba3de14 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:17.697995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59e7f39-a15b-4709-b75d-8a28dba3de14 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:17.762229 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179586ea-434c-4d8f-9d3a-1fd12475f172 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:17.762864 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179586ea-434c-4d8f-9d3a-1fd12475f172 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:17.763143 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179586ea-434c-4d8f-9d3a-1fd12475f172 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:17.763413 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179586ea-434c-4d8f-9d3a-1fd12475f172 admin admin] PATH_INFO: `/v3/projects/40a705cb20e140c49c92ec26fd4e95a7` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:17.764622 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179586ea-434c-4d8f-9d3a-1fd12475f172 admin admin] RBAC: Authorizing `identity:delete_project(project_id=40a705cb20e140c49c92ec26fd4e95a7)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:17.766538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179586ea-434c-4d8f-9d3a-1fd12475f172 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:17.819340 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-179586ea-434c-4d8f-9d3a-1fd12475f172 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '40a705cb20e140c49c92ec26fd4e95a7', 'request_id': 'req-179586ea-434c-4d8f-9d3a-1fd12475f172'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:17.829502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-179586ea-434c-4d8f-9d3a-1fd12475f172 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '40a705cb20e140c49c92ec26fd4e95a7', 'request_id': 'req-179586ea-434c-4d8f-9d3a-1fd12475f172'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:17.838231 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-179586ea-434c-4d8f-9d3a-1fd12475f172 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '40a705cb20e140c49c92ec26fd4e95a7', 'request_id': 'req-179586ea-434c-4d8f-9d3a-1fd12475f172'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:17.851160 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-179586ea-434c-4d8f-9d3a-1fd12475f172 admin admin] The token cache is being invalidate because project 40a705cb20e140c49c92ec26fd4e95a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96564) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:04:17.851523 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-179586ea-434c-4d8f-9d3a-1fd12475f172 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:17.852151 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-179586ea-434c-4d8f-9d3a-1fd12475f172 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:17.854743 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1233/2494] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:04:17 2022] DELETE /identity/v3/projects/40a705cb20e140c49c92ec26fd4e95a7 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:04:17.859478 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a1790d-5ae5-4da0-9fc0-ef16b251162f None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:17.859884 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a1790d-5ae5-4da0-9fc0-ef16b251162f None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:17.860180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a1790d-5ae5-4da0-9fc0-ef16b251162f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:17.882326 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-61a1790d-5ae5-4da0-9fc0-ef16b251162f None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:17.883638 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-61a1790d-5ae5-4da0-9fc0-ef16b251162f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:17.950762 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1262/2495] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:04:17.958841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c9feb3-e13b-4513-8bcb-cdc97a9b81ad None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:17.959928 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c9feb3-e13b-4513-8bcb-cdc97a9b81ad None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:17.960205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c9feb3-e13b-4513-8bcb-cdc97a9b81ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:17.983134 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-51c9feb3-e13b-4513-8bcb-cdc97a9b81ad None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:17.984937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-51c9feb3-e13b-4513-8bcb-cdc97a9b81ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:17.995955 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1234/2496] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:18.000377 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a1790d-5ae5-4da0-9fc0-ef16b251162f None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:18.011317 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a1790d-5ae5-4da0-9fc0-ef16b251162f None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:18.020594 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c9b49f-200e-48aa-8f50-ecfe4ae7b12f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:18.021204 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c9b49f-200e-48aa-8f50-ecfe4ae7b12f admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:18.021484 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c9b49f-200e-48aa-8f50-ecfe4ae7b12f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:18.021761 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c9b49f-200e-48aa-8f50-ecfe4ae7b12f admin admin] PATH_INFO: `/v3/domains` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:18.022405 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c9b49f-200e-48aa-8f50-ecfe4ae7b12f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96565) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:04:18.022788 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c9b49f-200e-48aa-8f50-ecfe4ae7b12f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:18.024417 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c9b49f-200e-48aa-8f50-ecfe4ae7b12f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:18.036483 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1263/2497] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:04:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:04:18.040407 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c9feb3-e13b-4513-8bcb-cdc97a9b81ad None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:18.045048 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:18.052592 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c9feb3-e13b-4513-8bcb-cdc97a9b81ad None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:18.058941 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc3bd80-2bc9-41d0-a77f-85a7e4e981ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:18.059916 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc3bd80-2bc9-41d0-a77f-85a7e4e981ae admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:18.060198 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc3bd80-2bc9-41d0-a77f-85a7e4e981ae admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:18.060470 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc3bd80-2bc9-41d0-a77f-85a7e4e981ae admin admin] PATH_INFO: `/v3/projects` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:18.061282 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc3bd80-2bc9-41d0-a77f-85a7e4e981ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:18.063205 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc3bd80-2bc9-41d0-a77f-85a7e4e981ae admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:18.093309 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1235/2498] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:04:18 2022] POST /identity/v3/projects => generated 377 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:04:18.097376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c9b49f-200e-48aa-8f50-ecfe4ae7b12f admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:18.107452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c9b49f-200e-48aa-8f50-ecfe4ae7b12f admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:18.139886 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:18.169820 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66736e41-576f-4add-955b-ce009bc8e790 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:18.170750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66736e41-576f-4add-955b-ce009bc8e790 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:18.171169 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66736e41-576f-4add-955b-ce009bc8e790 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:18.171445 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66736e41-576f-4add-955b-ce009bc8e790 admin admin] PATH_INFO: `/v3/users` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:18.172123 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66736e41-576f-4add-955b-ce009bc8e790 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:18.174256 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66736e41-576f-4add-955b-ce009bc8e790 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:18.199200 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1264/2499] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:04:18 2022] POST /identity/v3/users => generated 289 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:04:18.203929 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc3bd80-2bc9-41d0-a77f-85a7e4e981ae admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:18.217086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc3bd80-2bc9-41d0-a77f-85a7e4e981ae admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:18.224230 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e43d7d1-6f31-4586-976f-ec650c05d055 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:18.228794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e43d7d1-6f31-4586-976f-ec650c05d055 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:18.229510 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e43d7d1-6f31-4586-976f-ec650c05d055 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:18.229687 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e43d7d1-6f31-4586-976f-ec650c05d055 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:18.231938 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e43d7d1-6f31-4586-976f-ec650c05d055 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:18.243950 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e43d7d1-6f31-4586-976f-ec650c05d055 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:18.268099 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1236/2500] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:18 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:04:18.272488 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66736e41-576f-4add-955b-ce009bc8e790 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:18.287877 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66736e41-576f-4add-955b-ce009bc8e790 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:18.310596 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b918d2c-cd3f-4bf9-a6c4-6d8d70e8f55c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:18.312670 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b918d2c-cd3f-4bf9-a6c4-6d8d70e8f55c admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:18.313097 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b918d2c-cd3f-4bf9-a6c4-6d8d70e8f55c admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:18.313381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b918d2c-cd3f-4bf9-a6c4-6d8d70e8f55c admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:18.314041 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b918d2c-cd3f-4bf9-a6c4-6d8d70e8f55c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:18.316298 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b918d2c-cd3f-4bf9-a6c4-6d8d70e8f55c admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:18.338229 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1265/2501] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:18 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:04:18.342819 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e43d7d1-6f31-4586-976f-ec650c05d055 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:18.364812 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e43d7d1-6f31-4586-976f-ec650c05d055 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:18.377649 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634ca539-4212-4e6e-ac91-5cbd676d9f6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:18.379108 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634ca539-4212-4e6e-ac91-5cbd676d9f6e admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:18.379381 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634ca539-4212-4e6e-ac91-5cbd676d9f6e admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:18.379678 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634ca539-4212-4e6e-ac91-5cbd676d9f6e admin admin] PATH_INFO: `/v3/projects/0ee6f6e3ec0d48eeb81da86aa569e5da/users/4ea4ce54ec004545952613edcc3ff0c4/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:18.408481 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634ca539-4212-4e6e-ac91-5cbd676d9f6e admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ee6f6e3ec0d48eeb81da86aa569e5da, user_id=4ea4ce54ec004545952613edcc3ff0c4, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:18.412584 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634ca539-4212-4e6e-ac91-5cbd676d9f6e admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:18.444654 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1237/2502] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:04:18 2022] PUT /identity/v3/projects/0ee6f6e3ec0d48eeb81da86aa569e5da/users/4ea4ce54ec004545952613edcc3ff0c4/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:18.446502 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9410a60a-cd61-4a61-9e80-3ad64764b143 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:18.446786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9410a60a-cd61-4a61-9e80-3ad64764b143 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:18.447055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9410a60a-cd61-4a61-9e80-3ad64764b143 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:18.501901 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-9410a60a-cd61-4a61-9e80-3ad64764b143 None None] MFA Rules not processed for user `4ea4ce54ec004545952613edcc3ff0c4`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:18.504154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-9410a60a-cd61-4a61-9e80-3ad64764b143 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:18.619437 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1266/2503] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:18 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:18.631514 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634ca539-4212-4e6e-ac91-5cbd676d9f6e admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:18.633917 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-634ca539-4212-4e6e-ac91-5cbd676d9f6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:18.704457 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634ca539-4212-4e6e-ac91-5cbd676d9f6e admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:18.708791 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c874d4-2e66-4e9a-a104-dcc42f9ddfd9 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9808dcba0b64027bb70e31d1569957d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:18.709516 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c874d4-2e66-4e9a-a104-dcc42f9ddfd9 service glance] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:18.709896 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c874d4-2e66-4e9a-a104-dcc42f9ddfd9 service glance] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:18.710286 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c874d4-2e66-4e9a-a104-dcc42f9ddfd9 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:18.720833 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c874d4-2e66-4e9a-a104-dcc42f9ddfd9 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:18.723204 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c874d4-2e66-4e9a-a104-dcc42f9ddfd9 service glance] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:18.734707 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1238/2504] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:18 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:21.203815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9410a60a-cd61-4a61-9e80-3ad64764b143 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:21.205222 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-9410a60a-cd61-4a61-9e80-3ad64764b143 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:21.263124 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9410a60a-cd61-4a61-9e80-3ad64764b143 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:21.266634 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497c6621-6e9a-4f53-b770-d11a14562454 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:21.267236 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497c6621-6e9a-4f53-b770-d11a14562454 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:21.267514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497c6621-6e9a-4f53-b770-d11a14562454 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:21.267815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497c6621-6e9a-4f53-b770-d11a14562454 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:21.269769 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-497c6621-6e9a-4f53-b770-d11a14562454 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:21.323309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:21.338454 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497c6621-6e9a-4f53-b770-d11a14562454 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:21.338999 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497c6621-6e9a-4f53-b770-d11a14562454 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:21.348956 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1267/2505] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:21 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:28.729065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=49,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:31.344488 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=4 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:35.485849 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c874d4-2e66-4e9a-a104-dcc42f9ddfd9 service glance] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:35.487441 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-68c874d4-2e66-4e9a-a104-dcc42f9ddfd9 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:35.528119 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c874d4-2e66-4e9a-a104-dcc42f9ddfd9 service glance] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:35.531886 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0bf2bd-ee07-4b22-be2b-9c374924fc19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:35.532545 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0bf2bd-ee07-4b22-be2b-9c374924fc19 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:35.532827 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0bf2bd-ee07-4b22-be2b-9c374924fc19 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:35.533679 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0bf2bd-ee07-4b22-be2b-9c374924fc19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:35.535584 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-af0bf2bd-ee07-4b22-be2b-9c374924fc19 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:35.628378 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0bf2bd-ee07-4b22-be2b-9c374924fc19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:35.629799 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0bf2bd-ee07-4b22-be2b-9c374924fc19 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:35.639720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1239/2506] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:35 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:37.484569 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497c6621-6e9a-4f53-b770-d11a14562454 service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:37.486731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-497c6621-6e9a-4f53-b770-d11a14562454 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:37.674400 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497c6621-6e9a-4f53-b770-d11a14562454 service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:37.680091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea3289c-d378-4569-8b57-3f827185c090 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:37.680694 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea3289c-d378-4569-8b57-3f827185c090 service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:37.680970 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea3289c-d378-4569-8b57-3f827185c090 service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:37.681239 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea3289c-d378-4569-8b57-3f827185c090 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:37.682956 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-9ea3289c-d378-4569-8b57-3f827185c090 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:37.695539 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea3289c-d378-4569-8b57-3f827185c090 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:37.697220 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea3289c-d378-4569-8b57-3f827185c090 service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:37.708832 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1268/2507] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:38.672391 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0bf2bd-ee07-4b22-be2b-9c374924fc19 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:38.754072 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0bf2bd-ee07-4b22-be2b-9c374924fc19 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:38.758891 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39fe340-6aff-4e3a-84b2-23967a99b8fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:38.759502 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39fe340-6aff-4e3a-84b2-23967a99b8fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:38.759809 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39fe340-6aff-4e3a-84b2-23967a99b8fe admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:38.760083 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39fe340-6aff-4e3a-84b2-23967a99b8fe admin admin] PATH_INFO: `/v3/users/4ea4ce54ec004545952613edcc3ff0c4` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:38.761189 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39fe340-6aff-4e3a-84b2-23967a99b8fe admin admin] RBAC: Authorizing `identity:delete_user(user_id=4ea4ce54ec004545952613edcc3ff0c4)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:38.763107 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39fe340-6aff-4e3a-84b2-23967a99b8fe admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:38.796455 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:38.812964 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d39fe340-6aff-4e3a-84b2-23967a99b8fe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4ea4ce54ec004545952613edcc3ff0c4'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:38.821665 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d39fe340-6aff-4e3a-84b2-23967a99b8fe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4ea4ce54ec004545952613edcc3ff0c4'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:38.822472 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d39fe340-6aff-4e3a-84b2-23967a99b8fe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4ea4ce54ec004545952613edcc3ff0c4'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:38.823199 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d39fe340-6aff-4e3a-84b2-23967a99b8fe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4ea4ce54ec004545952613edcc3ff0c4'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:38.837182 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d39fe340-6aff-4e3a-84b2-23967a99b8fe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4ea4ce54ec004545952613edcc3ff0c4'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:38.853214 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1240/2508] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:04:38 2022] DELETE /identity/v3/users/4ea4ce54ec004545952613edcc3ff0c4 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:38.857834 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efac7dae-5a80-4861-af23-ddfd78642f1d None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:38.858309 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efac7dae-5a80-4861-af23-ddfd78642f1d None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:38.858892 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efac7dae-5a80-4861-af23-ddfd78642f1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:38.900732 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-efac7dae-5a80-4861-af23-ddfd78642f1d None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:38.901950 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-efac7dae-5a80-4861-af23-ddfd78642f1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:38.962422 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1269/2509] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:38.972257 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39fe340-6aff-4e3a-84b2-23967a99b8fe admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:38.973685 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-d39fe340-6aff-4e3a-84b2-23967a99b8fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:39.030207 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39fe340-6aff-4e3a-84b2-23967a99b8fe admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:39.034046 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb06c38e-0d58-452f-bb44-ee093e5347d7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:39.034638 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb06c38e-0d58-452f-bb44-ee093e5347d7 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:39.034921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb06c38e-0d58-452f-bb44-ee093e5347d7 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:39.035197 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb06c38e-0d58-452f-bb44-ee093e5347d7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:39.044244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb06c38e-0d58-452f-bb44-ee093e5347d7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:39.045666 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb06c38e-0d58-452f-bb44-ee093e5347d7 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:39.055747 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1241/2510] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:39.323741 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efac7dae-5a80-4861-af23-ddfd78642f1d None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:39.326687 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-efac7dae-5a80-4861-af23-ddfd78642f1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:39.336552 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:39.342311 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efac7dae-5a80-4861-af23-ddfd78642f1d None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:39.347299 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7b9d94-47ca-4cf2-925e-f230a90b24f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:39.348028 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b9d94-47ca-4cf2-925e-f230a90b24f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:39.348280 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b9d94-47ca-4cf2-925e-f230a90b24f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:39.348674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b9d94-47ca-4cf2-925e-f230a90b24f2 admin admin] PATH_INFO: `/v3/projects/0ee6f6e3ec0d48eeb81da86aa569e5da` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:39.349840 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7b9d94-47ca-4cf2-925e-f230a90b24f2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ee6f6e3ec0d48eeb81da86aa569e5da)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:39.351800 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7b9d94-47ca-4cf2-925e-f230a90b24f2 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:39.399083 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7a7b9d94-47ca-4cf2-925e-f230a90b24f2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ee6f6e3ec0d48eeb81da86aa569e5da', 'request_id': 'req-7a7b9d94-47ca-4cf2-925e-f230a90b24f2'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:39.404441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7a7b9d94-47ca-4cf2-925e-f230a90b24f2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ee6f6e3ec0d48eeb81da86aa569e5da', 'request_id': 'req-7a7b9d94-47ca-4cf2-925e-f230a90b24f2'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:39.411084 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7a7b9d94-47ca-4cf2-925e-f230a90b24f2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ee6f6e3ec0d48eeb81da86aa569e5da', 'request_id': 'req-7a7b9d94-47ca-4cf2-925e-f230a90b24f2'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:39.423301 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7a7b9d94-47ca-4cf2-925e-f230a90b24f2 admin admin] The token cache is being invalidate because project 0ee6f6e3ec0d48eeb81da86aa569e5da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:04:39.424308 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7a7b9d94-47ca-4cf2-925e-f230a90b24f2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:39.424946 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-7a7b9d94-47ca-4cf2-925e-f230a90b24f2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:39.426489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1270/2511] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:04:39 2022] DELETE /identity/v3/projects/0ee6f6e3ec0d48eeb81da86aa569e5da => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:39.432585 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526f6a6c-e1fa-4cc4-92a4-168595fd1507 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:39.432776 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526f6a6c-e1fa-4cc4-92a4-168595fd1507 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:39.433031 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526f6a6c-e1fa-4cc4-92a4-168595fd1507 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:39.482625 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-526f6a6c-e1fa-4cc4-92a4-168595fd1507 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:39.483878 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-526f6a6c-e1fa-4cc4-92a4-168595fd1507 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:39.543153 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1242/2512] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:39.552193 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fcf512-2cec-4188-ac1d-f5c6b99ca861 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:39.552473 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fcf512-2cec-4188-ac1d-f5c6b99ca861 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:39.552742 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fcf512-2cec-4188-ac1d-f5c6b99ca861 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:39.573114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-60fcf512-2cec-4188-ac1d-f5c6b99ca861 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:39.574235 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-60fcf512-2cec-4188-ac1d-f5c6b99ca861 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:39.582151 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1271/2513] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:39.586282 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526f6a6c-e1fa-4cc4-92a4-168595fd1507 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:39.596770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526f6a6c-e1fa-4cc4-92a4-168595fd1507 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:39.601393 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabb7341-e353-459f-af11-9fa2dc076a74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:39.601983 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabb7341-e353-459f-af11-9fa2dc076a74 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:39.602264 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabb7341-e353-459f-af11-9fa2dc076a74 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:39.602538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabb7341-e353-459f-af11-9fa2dc076a74 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:39.603181 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cabb7341-e353-459f-af11-9fa2dc076a74 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:04:39.603481 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cabb7341-e353-459f-af11-9fa2dc076a74 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:39.605107 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cabb7341-e353-459f-af11-9fa2dc076a74 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:39.616744 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1243/2514] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:04:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:04:39.620907 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fcf512-2cec-4188-ac1d-f5c6b99ca861 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:39.630488 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fcf512-2cec-4188-ac1d-f5c6b99ca861 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:39.635142 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6918ea25-0051-4ce3-b281-2eb12c2612c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:39.635771 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6918ea25-0051-4ce3-b281-2eb12c2612c6 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:39.636054 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6918ea25-0051-4ce3-b281-2eb12c2612c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:39.636329 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6918ea25-0051-4ce3-b281-2eb12c2612c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:39.636984 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6918ea25-0051-4ce3-b281-2eb12c2612c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:39.638855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6918ea25-0051-4ce3-b281-2eb12c2612c6 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:39.665544 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1272/2515] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:04:39 2022] POST /identity/v3/projects => generated 385 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:04:39.669541 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabb7341-e353-459f-af11-9fa2dc076a74 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:39.679388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabb7341-e353-459f-af11-9fa2dc076a74 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:39.688095 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:39.738910 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c272867-ad23-47d7-afbc-32641eb62d64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:39.739513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c272867-ad23-47d7-afbc-32641eb62d64 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:39.739837 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c272867-ad23-47d7-afbc-32641eb62d64 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:39.740116 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c272867-ad23-47d7-afbc-32641eb62d64 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:39.740744 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c272867-ad23-47d7-afbc-32641eb62d64 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:39.742664 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c272867-ad23-47d7-afbc-32641eb62d64 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:39.764077 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1244/2516] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:04:39 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:04:39.768529 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6918ea25-0051-4ce3-b281-2eb12c2612c6 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:39.778685 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6918ea25-0051-4ce3-b281-2eb12c2612c6 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:39.783859 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72b1247-9fc0-4016-a7e3-21679af7bc73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:39.784446 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72b1247-9fc0-4016-a7e3-21679af7bc73 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:39.784723 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72b1247-9fc0-4016-a7e3-21679af7bc73 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:39.784990 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72b1247-9fc0-4016-a7e3-21679af7bc73 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:39.785620 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72b1247-9fc0-4016-a7e3-21679af7bc73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:39.787187 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72b1247-9fc0-4016-a7e3-21679af7bc73 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:39.796807 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1273/2517] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:39 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:04:39.801204 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c272867-ad23-47d7-afbc-32641eb62d64 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:39.811173 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c272867-ad23-47d7-afbc-32641eb62d64 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:39.816571 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449a4ff9-a79c-4608-9a2c-7eb6f1e9849b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:39.817424 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449a4ff9-a79c-4608-9a2c-7eb6f1e9849b admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:39.817707 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449a4ff9-a79c-4608-9a2c-7eb6f1e9849b admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:39.817982 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449a4ff9-a79c-4608-9a2c-7eb6f1e9849b admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:39.818622 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449a4ff9-a79c-4608-9a2c-7eb6f1e9849b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:39.820259 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449a4ff9-a79c-4608-9a2c-7eb6f1e9849b admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:39.830978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1245/2518] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:39 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:04:39.834960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72b1247-9fc0-4016-a7e3-21679af7bc73 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:39.844873 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72b1247-9fc0-4016-a7e3-21679af7bc73 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:39.849548 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf94980-7a95-4fab-8bf5-de8d8e5b1892 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:39.850180 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf94980-7a95-4fab-8bf5-de8d8e5b1892 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:39.850457 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf94980-7a95-4fab-8bf5-de8d8e5b1892 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:39.850731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf94980-7a95-4fab-8bf5-de8d8e5b1892 admin admin] PATH_INFO: `/v3/projects/953f299bf80f44c69fe4453626c41290/users/902a5bc3be0444f3939c7c363733f18c/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:39.867398 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf94980-7a95-4fab-8bf5-de8d8e5b1892 admin admin] RBAC: Authorizing `identity:create_grant(project_id=953f299bf80f44c69fe4453626c41290, user_id=902a5bc3be0444f3939c7c363733f18c, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:39.870036 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf94980-7a95-4fab-8bf5-de8d8e5b1892 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:39.885600 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1274/2519] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:04:39 2022] PUT /identity/v3/projects/953f299bf80f44c69fe4453626c41290/users/902a5bc3be0444f3939c7c363733f18c/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:39.896420 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7186cf4-6ffd-40e1-91d6-32d10b0cde0a None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:39.896704 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7186cf4-6ffd-40e1-91d6-32d10b0cde0a None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:39.896979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7186cf4-6ffd-40e1-91d6-32d10b0cde0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:39.925927 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-e7186cf4-6ffd-40e1-91d6-32d10b0cde0a None None] MFA Rules not processed for user `902a5bc3be0444f3939c7c363733f18c`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:39.927152 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e7186cf4-6ffd-40e1-91d6-32d10b0cde0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:39.991329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1246/2520] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:39 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:40.003091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf94980-7a95-4fab-8bf5-de8d8e5b1892 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:40.004859 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-daf94980-7a95-4fab-8bf5-de8d8e5b1892 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:40.060944 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf94980-7a95-4fab-8bf5-de8d8e5b1892 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:40.065378 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5456cf8a-1502-4027-9553-5036a2c12840 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:40.065978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5456cf8a-1502-4027-9553-5036a2c12840 service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:40.066255 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5456cf8a-1502-4027-9553-5036a2c12840 service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:40.066526 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5456cf8a-1502-4027-9553-5036a2c12840 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:40.075045 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5456cf8a-1502-4027-9553-5036a2c12840 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:40.076501 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5456cf8a-1502-4027-9553-5036a2c12840 service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:40.085484 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1275/2521] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:40 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:42.633027 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7186cf4-6ffd-40e1-91d6-32d10b0cde0a None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:42.694455 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7186cf4-6ffd-40e1-91d6-32d10b0cde0a None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:42.699038 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618e95fd-28e8-46df-b975-c24497600be9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:42.699695 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618e95fd-28e8-46df-b975-c24497600be9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:42.699979 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618e95fd-28e8-46df-b975-c24497600be9 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:42.700252 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618e95fd-28e8-46df-b975-c24497600be9 admin admin] PATH_INFO: `/v3/users/902a5bc3be0444f3939c7c363733f18c` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:42.701363 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618e95fd-28e8-46df-b975-c24497600be9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=902a5bc3be0444f3939c7c363733f18c)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:42.703269 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618e95fd-28e8-46df-b975-c24497600be9 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:42.726025 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:42.750483 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-618e95fd-28e8-46df-b975-c24497600be9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '902a5bc3be0444f3939c7c363733f18c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:42.757809 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-618e95fd-28e8-46df-b975-c24497600be9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '902a5bc3be0444f3939c7c363733f18c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:42.758458 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-618e95fd-28e8-46df-b975-c24497600be9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '902a5bc3be0444f3939c7c363733f18c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:42.759061 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-618e95fd-28e8-46df-b975-c24497600be9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '902a5bc3be0444f3939c7c363733f18c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:42.771275 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-618e95fd-28e8-46df-b975-c24497600be9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '902a5bc3be0444f3939c7c363733f18c'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:42.786437 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1247/2522] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:04:42 2022] DELETE /identity/v3/users/902a5bc3be0444f3939c7c363733f18c => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:42.791108 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc6084c-220e-4b62-8880-cdfb01580b81 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:42.791623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc6084c-220e-4b62-8880-cdfb01580b81 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:42.791723 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc6084c-220e-4b62-8880-cdfb01580b81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:42.811451 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-0dc6084c-220e-4b62-8880-cdfb01580b81 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:42.812662 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0dc6084c-220e-4b62-8880-cdfb01580b81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:42.833805 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:42.876751 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1276/2523] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:04:42.886900 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618e95fd-28e8-46df-b975-c24497600be9 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:42.888327 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-618e95fd-28e8-46df-b975-c24497600be9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:42.945862 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618e95fd-28e8-46df-b975-c24497600be9 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:42.949636 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6547e138-8446-4678-a755-27c733c889d7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:42.950237 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6547e138-8446-4678-a755-27c733c889d7 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:42.950513 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6547e138-8446-4678-a755-27c733c889d7 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:42.950786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6547e138-8446-4678-a755-27c733c889d7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:42.960507 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6547e138-8446-4678-a755-27c733c889d7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:42.961954 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6547e138-8446-4678-a755-27c733c889d7 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:42.971907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1248/2524] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:43.248434 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc6084c-220e-4b62-8880-cdfb01580b81 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:43.249945 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0dc6084c-220e-4b62-8880-cdfb01580b81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:43.267422 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc6084c-220e-4b62-8880-cdfb01580b81 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:43.272971 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aec82d-ff22-478a-b6e0-58cfba7d8a97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:43.273712 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aec82d-ff22-478a-b6e0-58cfba7d8a97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:43.274182 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aec82d-ff22-478a-b6e0-58cfba7d8a97 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:43.274448 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aec82d-ff22-478a-b6e0-58cfba7d8a97 admin admin] PATH_INFO: `/v3/projects/953f299bf80f44c69fe4453626c41290` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:43.275699 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74aec82d-ff22-478a-b6e0-58cfba7d8a97 admin admin] RBAC: Authorizing `identity:delete_project(project_id=953f299bf80f44c69fe4453626c41290)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:43.277735 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74aec82d-ff22-478a-b6e0-58cfba7d8a97 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:43.334476 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-74aec82d-ff22-478a-b6e0-58cfba7d8a97 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '953f299bf80f44c69fe4453626c41290', 'request_id': 'req-74aec82d-ff22-478a-b6e0-58cfba7d8a97'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:43.340999 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-74aec82d-ff22-478a-b6e0-58cfba7d8a97 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '953f299bf80f44c69fe4453626c41290', 'request_id': 'req-74aec82d-ff22-478a-b6e0-58cfba7d8a97'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:43.347782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-74aec82d-ff22-478a-b6e0-58cfba7d8a97 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '953f299bf80f44c69fe4453626c41290', 'request_id': 'req-74aec82d-ff22-478a-b6e0-58cfba7d8a97'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:43.361756 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-74aec82d-ff22-478a-b6e0-58cfba7d8a97 admin admin] The token cache is being invalidate because project 953f299bf80f44c69fe4453626c41290 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:04:43.362112 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-74aec82d-ff22-478a-b6e0-58cfba7d8a97 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:43.362918 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-74aec82d-ff22-478a-b6e0-58cfba7d8a97 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:04:43.364523 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1277/2525] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:04:43 2022] DELETE /identity/v3/projects/953f299bf80f44c69fe4453626c41290 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:43.369778 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce99a21a-ec23-4269-98b1-f77b0011123e None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:43.370068 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce99a21a-ec23-4269-98b1-f77b0011123e None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:43.370352 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce99a21a-ec23-4269-98b1-f77b0011123e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:43.392567 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-ce99a21a-ec23-4269-98b1-f77b0011123e None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:43.393784 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ce99a21a-ec23-4269-98b1-f77b0011123e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:43.464010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1249/2526] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:43.472542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c192095-f9a5-4adf-85cb-1db4ad40e9fe None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:43.472816 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c192095-f9a5-4adf-85cb-1db4ad40e9fe None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:43.473089 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c192095-f9a5-4adf-85cb-1db4ad40e9fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:43.493728 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-5c192095-f9a5-4adf-85cb-1db4ad40e9fe None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:43.494937 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5c192095-f9a5-4adf-85cb-1db4ad40e9fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:43.504086 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1278/2527] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:43.507935 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce99a21a-ec23-4269-98b1-f77b0011123e None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:43.518965 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce99a21a-ec23-4269-98b1-f77b0011123e None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:43.523957 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12dd3c80-b40c-4ad9-aef5-01c449ae0cac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:43.524576 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dd3c80-b40c-4ad9-aef5-01c449ae0cac admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:43.524863 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dd3c80-b40c-4ad9-aef5-01c449ae0cac admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:43.525140 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dd3c80-b40c-4ad9-aef5-01c449ae0cac admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:43.525799 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12dd3c80-b40c-4ad9-aef5-01c449ae0cac admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:04:43.526105 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12dd3c80-b40c-4ad9-aef5-01c449ae0cac admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:43.527733 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12dd3c80-b40c-4ad9-aef5-01c449ae0cac admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:43.541061 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1250/2528] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:04:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:04:43.545500 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c192095-f9a5-4adf-85cb-1db4ad40e9fe None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:43.555294 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c192095-f9a5-4adf-85cb-1db4ad40e9fe None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:43.560047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0c0b25-666a-4ec5-9216-3adbd8280e7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:43.560675 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0c0b25-666a-4ec5-9216-3adbd8280e7e admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:43.560955 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0c0b25-666a-4ec5-9216-3adbd8280e7e admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:43.561227 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0c0b25-666a-4ec5-9216-3adbd8280e7e admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:43.561856 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0c0b25-666a-4ec5-9216-3adbd8280e7e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:43.563828 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0c0b25-666a-4ec5-9216-3adbd8280e7e admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:43.591605 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1279/2529] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:04:43 2022] POST /identity/v3/projects => generated 365 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:04:43.596370 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12dd3c80-b40c-4ad9-aef5-01c449ae0cac admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:43.606249 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12dd3c80-b40c-4ad9-aef5-01c449ae0cac admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:43.635452 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:43.668588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c323c1-1937-4afd-b42c-fc7ec0ea0004 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:43.669191 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c323c1-1937-4afd-b42c-fc7ec0ea0004 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:43.669475 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c323c1-1937-4afd-b42c-fc7ec0ea0004 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:43.669750 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c323c1-1937-4afd-b42c-fc7ec0ea0004 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:43.670386 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c323c1-1937-4afd-b42c-fc7ec0ea0004 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:43.672328 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c323c1-1937-4afd-b42c-fc7ec0ea0004 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:43.693307 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1251/2530] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:04:43 2022] POST /identity/v3/users => generated 283 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:04:43.697392 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0c0b25-666a-4ec5-9216-3adbd8280e7e admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:43.708441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0c0b25-666a-4ec5-9216-3adbd8280e7e admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:43.713102 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82b87c8-8d6d-4bdc-8bd2-50fd31158750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:43.713692 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82b87c8-8d6d-4bdc-8bd2-50fd31158750 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:43.713972 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82b87c8-8d6d-4bdc-8bd2-50fd31158750 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:43.714286 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82b87c8-8d6d-4bdc-8bd2-50fd31158750 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:43.714939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82b87c8-8d6d-4bdc-8bd2-50fd31158750 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:43.716640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82b87c8-8d6d-4bdc-8bd2-50fd31158750 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:43.727658 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1280/2531] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:04:43.731655 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c323c1-1937-4afd-b42c-fc7ec0ea0004 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:43.741792 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c323c1-1937-4afd-b42c-fc7ec0ea0004 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:43.746530 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420436b6-8c7a-4f1b-bfac-f51c3a617c44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:43.747123 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420436b6-8c7a-4f1b-bfac-f51c3a617c44 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:43.747402 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420436b6-8c7a-4f1b-bfac-f51c3a617c44 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:43.747705 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420436b6-8c7a-4f1b-bfac-f51c3a617c44 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:43.748342 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420436b6-8c7a-4f1b-bfac-f51c3a617c44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:43.749931 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420436b6-8c7a-4f1b-bfac-f51c3a617c44 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:43.759878 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1252/2532] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:04:43 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:04:43.764079 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82b87c8-8d6d-4bdc-8bd2-50fd31158750 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:43.774975 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82b87c8-8d6d-4bdc-8bd2-50fd31158750 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:43.779719 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a14128d-4a73-4ad8-947f-a948986839e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:43.780367 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a14128d-4a73-4ad8-947f-a948986839e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:43.780649 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a14128d-4a73-4ad8-947f-a948986839e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:43.780925 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a14128d-4a73-4ad8-947f-a948986839e1 admin admin] PATH_INFO: `/v3/projects/f09b0b91abaf46dea395453d35a54749/users/7306f644279b4a8bb52dfd40d7fa6bdd/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:43.795784 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a14128d-4a73-4ad8-947f-a948986839e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f09b0b91abaf46dea395453d35a54749, user_id=7306f644279b4a8bb52dfd40d7fa6bdd, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:43.799300 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a14128d-4a73-4ad8-947f-a948986839e1 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:43.813017 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1281/2533] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:04:43 2022] PUT /identity/v3/projects/f09b0b91abaf46dea395453d35a54749/users/7306f644279b4a8bb52dfd40d7fa6bdd/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:04:43.879232 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34623869-7dee-4cd6-95e5-f05878ee5669 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:43.879517 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34623869-7dee-4cd6-95e5-f05878ee5669 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:43.879827 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34623869-7dee-4cd6-95e5-f05878ee5669 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:43.907061 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-34623869-7dee-4cd6-95e5-f05878ee5669 None None] MFA Rules not processed for user `7306f644279b4a8bb52dfd40d7fa6bdd`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:04:43.908818 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-34623869-7dee-4cd6-95e5-f05878ee5669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:43.972002 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1253/2534] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:04:43 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:04:43.983211 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a14128d-4a73-4ad8-947f-a948986839e1 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:43.984908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-5a14128d-4a73-4ad8-947f-a948986839e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:44.029711 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:44.042292 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a14128d-4a73-4ad8-947f-a948986839e1 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:44.046219 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc06f7d0-5391-4173-88ee-fb5258f67f4f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:44.046815 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc06f7d0-5391-4173-88ee-fb5258f67f4f service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:44.047091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc06f7d0-5391-4173-88ee-fb5258f67f4f service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:44.047363 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc06f7d0-5391-4173-88ee-fb5258f67f4f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:44.056145 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc06f7d0-5391-4173-88ee-fb5258f67f4f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:44.057589 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc06f7d0-5391-4173-88ee-fb5258f67f4f service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:44.066969 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1282/2535] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:43 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:44.705220 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34623869-7dee-4cd6-95e5-f05878ee5669 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:04:44.706658 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-34623869-7dee-4cd6-95e5-f05878ee5669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:44.803532 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34623869-7dee-4cd6-95e5-f05878ee5669 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:04:44.808606 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12d819f-19f8-4cde-8a20-89bd93710cd6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70ea2aaf529b46dabb70b71f4dcc3860', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:04:44.809203 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12d819f-19f8-4cde-8a20-89bd93710cd6 service placement] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:04:44.809481 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12d819f-19f8-4cde-8a20-89bd93710cd6 service placement] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:04:44.809752 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12d819f-19f8-4cde-8a20-89bd93710cd6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:04:44.811456 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-b12d819f-19f8-4cde-8a20-89bd93710cd6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:04:44.823380 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12d819f-19f8-4cde-8a20-89bd93710cd6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:04:44.824978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12d819f-19f8-4cde-8a20-89bd93710cd6 service placement] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:04:44.834759 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1254/2536] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:04:44 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:04:54.062319 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=6 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:04:54.830601 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=93,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:05:21.817121 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbae7d58-31f5-4969-aa72-462328f90e85 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:05:21.817681 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbae7d58-31f5-4969-aa72-462328f90e85 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:05:21.817681 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbae7d58-31f5-4969-aa72-462328f90e85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:05:21.837201 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-dbae7d58-31f5-4969-aa72-462328f90e85 None None] MFA Rules not processed for user `7b9ebf0c707045908f3950c7077b3554`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:05:21.838388 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-dbae7d58-31f5-4969-aa72-462328f90e85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:05:21.892644 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1283/2537] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:05:21 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:05:27.688235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12d819f-19f8-4cde-8a20-89bd93710cd6 service placement] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:05:27.751312 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12d819f-19f8-4cde-8a20-89bd93710cd6 service placement] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:05:27.755986 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402a4a1a-1496-4fa6-a6d3-daac645d64a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:05:27.756614 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402a4a1a-1496-4fa6-a6d3-daac645d64a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:05:27.756895 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402a4a1a-1496-4fa6-a6d3-daac645d64a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:05:27.757172 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402a4a1a-1496-4fa6-a6d3-daac645d64a0 admin admin] PATH_INFO: `/v3/users/7306f644279b4a8bb52dfd40d7fa6bdd` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:05:27.758290 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402a4a1a-1496-4fa6-a6d3-daac645d64a0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7306f644279b4a8bb52dfd40d7fa6bdd)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:05:27.760260 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402a4a1a-1496-4fa6-a6d3-daac645d64a0 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:05:27.806574 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-402a4a1a-1496-4fa6-a6d3-daac645d64a0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7306f644279b4a8bb52dfd40d7fa6bdd'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:05:27.813544 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-402a4a1a-1496-4fa6-a6d3-daac645d64a0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7306f644279b4a8bb52dfd40d7fa6bdd'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:05:27.814244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-402a4a1a-1496-4fa6-a6d3-daac645d64a0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7306f644279b4a8bb52dfd40d7fa6bdd'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:05:27.814898 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-402a4a1a-1496-4fa6-a6d3-daac645d64a0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7306f644279b4a8bb52dfd40d7fa6bdd'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:05:27.828076 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-402a4a1a-1496-4fa6-a6d3-daac645d64a0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7306f644279b4a8bb52dfd40d7fa6bdd'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:05:27.840437 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1255/2538] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:05:27 2022] DELETE /identity/v3/users/7306f644279b4a8bb52dfd40d7fa6bdd => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:05:27.845525 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce2a1ee-01d7-468e-acd7-59641dbbfd47 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:05:27.846006 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce2a1ee-01d7-468e-acd7-59641dbbfd47 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:05:27.846123 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce2a1ee-01d7-468e-acd7-59641dbbfd47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:05:27.864789 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-0ce2a1ee-01d7-468e-acd7-59641dbbfd47 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:05:27.865988 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0ce2a1ee-01d7-468e-acd7-59641dbbfd47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:05:27.923746 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1284/2539] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:05:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:05:27.933430 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402a4a1a-1496-4fa6-a6d3-daac645d64a0 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:05:27.934830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-402a4a1a-1496-4fa6-a6d3-daac645d64a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:05:28.012549 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402a4a1a-1496-4fa6-a6d3-daac645d64a0 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:05:28.016826 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2932ee-3b9e-4a5c-b9fe-f574c4db4ba1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:05:28.017421 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2932ee-3b9e-4a5c-b9fe-f574c4db4ba1 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:05:28.017698 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2932ee-3b9e-4a5c-b9fe-f574c4db4ba1 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:05:28.017967 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2932ee-3b9e-4a5c-b9fe-f574c4db4ba1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:05:28.027046 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2932ee-3b9e-4a5c-b9fe-f574c4db4ba1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:05:28.028536 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2932ee-3b9e-4a5c-b9fe-f574c4db4ba1 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:05:28.038629 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1256/2540] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:05:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:05:28.305667 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce2a1ee-01d7-468e-acd7-59641dbbfd47 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:05:28.307160 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0ce2a1ee-01d7-468e-acd7-59641dbbfd47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:05:28.320091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce2a1ee-01d7-468e-acd7-59641dbbfd47 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:05:28.324828 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220f743a-aff7-4590-aa0d-cd4c7d75034f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:05:28.325533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220f743a-aff7-4590-aa0d-cd4c7d75034f admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:05:28.325814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220f743a-aff7-4590-aa0d-cd4c7d75034f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:05:28.326089 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220f743a-aff7-4590-aa0d-cd4c7d75034f admin admin] PATH_INFO: `/v3/projects/f09b0b91abaf46dea395453d35a54749` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:05:28.327321 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220f743a-aff7-4590-aa0d-cd4c7d75034f admin admin] RBAC: Authorizing `identity:delete_project(project_id=f09b0b91abaf46dea395453d35a54749)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:05:28.329859 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220f743a-aff7-4590-aa0d-cd4c7d75034f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:05:28.377763 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-220f743a-aff7-4590-aa0d-cd4c7d75034f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f09b0b91abaf46dea395453d35a54749', 'request_id': 'req-220f743a-aff7-4590-aa0d-cd4c7d75034f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:05:28.383051 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-220f743a-aff7-4590-aa0d-cd4c7d75034f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f09b0b91abaf46dea395453d35a54749', 'request_id': 'req-220f743a-aff7-4590-aa0d-cd4c7d75034f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:05:28.389147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-220f743a-aff7-4590-aa0d-cd4c7d75034f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f09b0b91abaf46dea395453d35a54749', 'request_id': 'req-220f743a-aff7-4590-aa0d-cd4c7d75034f'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:05:28.401736 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-220f743a-aff7-4590-aa0d-cd4c7d75034f admin admin] The token cache is being invalidate because project f09b0b91abaf46dea395453d35a54749 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:05:28.402045 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-220f743a-aff7-4590-aa0d-cd4c7d75034f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:05:28.402641 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-220f743a-aff7-4590-aa0d-cd4c7d75034f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:05:28.404137 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1285/2541] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:05:28 2022] DELETE /identity/v3/projects/f09b0b91abaf46dea395453d35a54749 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:05:38.034359 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:05:38.398782 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=79,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:06:17.549794 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2932ee-3b9e-4a5c-b9fe-f574c4db4ba1 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:17.551300 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-7f2932ee-3b9e-4a5c-b9fe-f574c4db4ba1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:17.564841 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2932ee-3b9e-4a5c-b9fe-f574c4db4ba1 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:17.621941 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0189a48-6c1b-4ee7-ac1a-1a1489d74bc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:17.622571 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0189a48-6c1b-4ee7-ac1a-1a1489d74bc8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:17.622853 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0189a48-6c1b-4ee7-ac1a-1a1489d74bc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:17.623158 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0189a48-6c1b-4ee7-ac1a-1a1489d74bc8 admin admin] PATH_INFO: `/v3/users/7b9ebf0c707045908f3950c7077b3554` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:17.624320 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0189a48-6c1b-4ee7-ac1a-1a1489d74bc8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b9ebf0c707045908f3950c7077b3554)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:17.626235 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0189a48-6c1b-4ee7-ac1a-1a1489d74bc8 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:17.671497 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d0189a48-6c1b-4ee7-ac1a-1a1489d74bc8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b9ebf0c707045908f3950c7077b3554'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:17.678115 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d0189a48-6c1b-4ee7-ac1a-1a1489d74bc8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b9ebf0c707045908f3950c7077b3554'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:17.678968 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d0189a48-6c1b-4ee7-ac1a-1a1489d74bc8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b9ebf0c707045908f3950c7077b3554'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:17.679612 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d0189a48-6c1b-4ee7-ac1a-1a1489d74bc8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b9ebf0c707045908f3950c7077b3554'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:17.691039 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-d0189a48-6c1b-4ee7-ac1a-1a1489d74bc8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b9ebf0c707045908f3950c7077b3554'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:17.703999 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1257/2542] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:06:17 2022] DELETE /identity/v3/users/7b9ebf0c707045908f3950c7077b3554 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:06:17.876564 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220f743a-aff7-4590-aa0d-cd4c7d75034f admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:17.878065 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-220f743a-aff7-4590-aa0d-cd4c7d75034f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:17.891502 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220f743a-aff7-4590-aa0d-cd4c7d75034f admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:17.950671 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51372bdf-2d53-4d00-a7c2-88b42d823825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:17.951223 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51372bdf-2d53-4d00-a7c2-88b42d823825 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:17.951502 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51372bdf-2d53-4d00-a7c2-88b42d823825 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:17.951805 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51372bdf-2d53-4d00-a7c2-88b42d823825 admin admin] PATH_INFO: `/v3/projects/3424a823161e49029c454a8818f7bd7f` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:17.952943 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51372bdf-2d53-4d00-a7c2-88b42d823825 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3424a823161e49029c454a8818f7bd7f)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:17.954828 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51372bdf-2d53-4d00-a7c2-88b42d823825 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:18.000786 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-51372bdf-2d53-4d00-a7c2-88b42d823825 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3424a823161e49029c454a8818f7bd7f', 'request_id': 'req-51372bdf-2d53-4d00-a7c2-88b42d823825'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:18.006167 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-51372bdf-2d53-4d00-a7c2-88b42d823825 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3424a823161e49029c454a8818f7bd7f', 'request_id': 'req-51372bdf-2d53-4d00-a7c2-88b42d823825'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:18.014056 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-51372bdf-2d53-4d00-a7c2-88b42d823825 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3424a823161e49029c454a8818f7bd7f', 'request_id': 'req-51372bdf-2d53-4d00-a7c2-88b42d823825'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:18.027229 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-51372bdf-2d53-4d00-a7c2-88b42d823825 admin admin] The token cache is being invalidate because project 3424a823161e49029c454a8818f7bd7f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:06:18.027514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-51372bdf-2d53-4d00-a7c2-88b42d823825 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:18.028157 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-51372bdf-2d53-4d00-a7c2-88b42d823825 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:18.029636 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1286/2543] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:06:17 2022] DELETE /identity/v3/projects/3424a823161e49029c454a8818f7bd7f => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:06:18.034665 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa0a560-16f7-4e6e-be17-8b5226ed96f1 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:18.034959 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa0a560-16f7-4e6e-be17-8b5226ed96f1 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:18.035265 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa0a560-16f7-4e6e-be17-8b5226ed96f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:18.055860 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-2fa0a560-16f7-4e6e-be17-8b5226ed96f1 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:06:18.057080 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-2fa0a560-16f7-4e6e-be17-8b5226ed96f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:18.117134 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1258/2544] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:06:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 31 15:06:18.124926 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eb0c47-385e-4462-a20a-af8f590e3ade None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:18.125212 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eb0c47-385e-4462-a20a-af8f590e3ade None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:18.125489 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eb0c47-385e-4462-a20a-af8f590e3ade None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:18.145091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-66eb0c47-385e-4462-a20a-af8f590e3ade None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:06:18.146278 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-66eb0c47-385e-4462-a20a-af8f590e3ade None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:18.154796 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1287/2545] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:06:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:06:18.158837 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa0a560-16f7-4e6e-be17-8b5226ed96f1 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:18.169111 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa0a560-16f7-4e6e-be17-8b5226ed96f1 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:18.173738 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc69fa5-757f-442e-9bcd-21e7dab31a34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:18.174352 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc69fa5-757f-442e-9bcd-21e7dab31a34 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:18.174633 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc69fa5-757f-442e-9bcd-21e7dab31a34 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:18.174906 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc69fa5-757f-442e-9bcd-21e7dab31a34 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:18.175546 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc69fa5-757f-442e-9bcd-21e7dab31a34 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:06:18.175874 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc69fa5-757f-442e-9bcd-21e7dab31a34 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:18.177465 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc69fa5-757f-442e-9bcd-21e7dab31a34 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:18.189202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1259/2546] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:06:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:06:18.193493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66eb0c47-385e-4462-a20a-af8f590e3ade None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:18.203187 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66eb0c47-385e-4462-a20a-af8f590e3ade None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:18.207762 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efcce26-4288-448f-8802-623a95c8c8b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:18.208349 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efcce26-4288-448f-8802-623a95c8c8b8 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:18.208623 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efcce26-4288-448f-8802-623a95c8c8b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:18.208892 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efcce26-4288-448f-8802-623a95c8c8b8 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:18.209524 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6efcce26-4288-448f-8802-623a95c8c8b8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:18.212086 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6efcce26-4288-448f-8802-623a95c8c8b8 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:18.238006 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1288/2547] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:06:18 2022] POST /identity/v3/projects => generated 399 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:06:18.242229 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc69fa5-757f-442e-9bcd-21e7dab31a34 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:18.252117 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc69fa5-757f-442e-9bcd-21e7dab31a34 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:18.255644 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:06:18.310172 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30261bb-13ad-4535-8c5a-c82fa9a1cf42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:18.310772 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30261bb-13ad-4535-8c5a-c82fa9a1cf42 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:18.311054 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30261bb-13ad-4535-8c5a-c82fa9a1cf42 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:18.311329 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30261bb-13ad-4535-8c5a-c82fa9a1cf42 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:18.311986 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30261bb-13ad-4535-8c5a-c82fa9a1cf42 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:18.313891 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30261bb-13ad-4535-8c5a-c82fa9a1cf42 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:18.335433 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1260/2548] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:06:18 2022] POST /identity/v3/users => generated 300 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:06:18.339914 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efcce26-4288-448f-8802-623a95c8c8b8 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:18.350106 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efcce26-4288-448f-8802-623a95c8c8b8 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:18.354725 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fbc212-6bb3-4d41-a8a5-2455e5a66d66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:18.355317 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fbc212-6bb3-4d41-a8a5-2455e5a66d66 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:18.355631 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fbc212-6bb3-4d41-a8a5-2455e5a66d66 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:18.355908 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fbc212-6bb3-4d41-a8a5-2455e5a66d66 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:18.356542 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fbc212-6bb3-4d41-a8a5-2455e5a66d66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:18.358115 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fbc212-6bb3-4d41-a8a5-2455e5a66d66 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:18.367849 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1289/2549] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:06:18 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:06:18.371709 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30261bb-13ad-4535-8c5a-c82fa9a1cf42 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:18.382648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30261bb-13ad-4535-8c5a-c82fa9a1cf42 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:18.387517 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6536b97e-2254-4ed2-8e1f-f35fd0a6de83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:18.388133 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6536b97e-2254-4ed2-8e1f-f35fd0a6de83 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:18.388411 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6536b97e-2254-4ed2-8e1f-f35fd0a6de83 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:18.388682 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6536b97e-2254-4ed2-8e1f-f35fd0a6de83 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:18.389353 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6536b97e-2254-4ed2-8e1f-f35fd0a6de83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:18.390936 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6536b97e-2254-4ed2-8e1f-f35fd0a6de83 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:18.402062 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1261/2550] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:06:18 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:06:18.406044 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fbc212-6bb3-4d41-a8a5-2455e5a66d66 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:18.415914 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fbc212-6bb3-4d41-a8a5-2455e5a66d66 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:18.420481 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00403ed9-0aab-456d-ba2e-f406ebef8987 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:18.421110 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00403ed9-0aab-456d-ba2e-f406ebef8987 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:18.421388 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00403ed9-0aab-456d-ba2e-f406ebef8987 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:18.421660 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00403ed9-0aab-456d-ba2e-f406ebef8987 admin admin] PATH_INFO: `/v3/projects/9fe5bc9b3d0b423dab3a2d6aba289afa/users/61c621e005994ab888b36c4799b99f09/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:18.435814 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00403ed9-0aab-456d-ba2e-f406ebef8987 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9fe5bc9b3d0b423dab3a2d6aba289afa, user_id=61c621e005994ab888b36c4799b99f09, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:18.438421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00403ed9-0aab-456d-ba2e-f406ebef8987 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:18.451779 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1290/2551] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:06:18 2022] PUT /identity/v3/projects/9fe5bc9b3d0b423dab3a2d6aba289afa/users/61c621e005994ab888b36c4799b99f09/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:06:18.462955 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854ad947-324e-4e9c-8ea5-a2b3dcbaf119 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:18.463186 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854ad947-324e-4e9c-8ea5-a2b3dcbaf119 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:18.463486 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854ad947-324e-4e9c-8ea5-a2b3dcbaf119 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:18.490520 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-854ad947-324e-4e9c-8ea5-a2b3dcbaf119 None None] MFA Rules not processed for user `61c621e005994ab888b36c4799b99f09`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:06:18.491761 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-854ad947-324e-4e9c-8ea5-a2b3dcbaf119 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:18.556596 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1262/2552] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:06:18 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:06:18.568231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00403ed9-0aab-456d-ba2e-f406ebef8987 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:18.570934 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-00403ed9-0aab-456d-ba2e-f406ebef8987 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:18.602333 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:06:18.651792 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00403ed9-0aab-456d-ba2e-f406ebef8987 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:18.655426 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27011b72-d779-4ce3-acc0-4477e44e354c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:18.656055 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27011b72-d779-4ce3-acc0-4477e44e354c service cinder] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:18.656334 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27011b72-d779-4ce3-acc0-4477e44e354c service cinder] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:18.656606 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27011b72-d779-4ce3-acc0-4477e44e354c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:18.664847 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27011b72-d779-4ce3-acc0-4477e44e354c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:18.666258 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27011b72-d779-4ce3-acc0-4477e44e354c service cinder] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:18.674887 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1291/2553] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:06:18 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:06:25.719623 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854ad947-324e-4e9c-8ea5-a2b3dcbaf119 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:25.779915 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854ad947-324e-4e9c-8ea5-a2b3dcbaf119 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:25.784438 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b25296f-8828-4192-97be-84092290ff97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:25.784993 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b25296f-8828-4192-97be-84092290ff97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:25.785269 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b25296f-8828-4192-97be-84092290ff97 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:25.785538 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b25296f-8828-4192-97be-84092290ff97 admin admin] PATH_INFO: `/v3/users/61c621e005994ab888b36c4799b99f09` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:25.786631 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b25296f-8828-4192-97be-84092290ff97 admin admin] RBAC: Authorizing `identity:delete_user(user_id=61c621e005994ab888b36c4799b99f09)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:25.789066 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b25296f-8828-4192-97be-84092290ff97 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:25.804881 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:06:25.835458 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5b25296f-8828-4192-97be-84092290ff97 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '61c621e005994ab888b36c4799b99f09'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:25.842338 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5b25296f-8828-4192-97be-84092290ff97 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '61c621e005994ab888b36c4799b99f09'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:25.843016 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5b25296f-8828-4192-97be-84092290ff97 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '61c621e005994ab888b36c4799b99f09'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:25.843646 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5b25296f-8828-4192-97be-84092290ff97 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '61c621e005994ab888b36c4799b99f09'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:25.856219 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-5b25296f-8828-4192-97be-84092290ff97 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '61c621e005994ab888b36c4799b99f09'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:25.869692 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1263/2554] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:06:25 2022] DELETE /identity/v3/users/61c621e005994ab888b36c4799b99f09 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:06:25.874184 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b04a796-0b1d-4a4c-ae47-599c732a9635 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:25.874640 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b04a796-0b1d-4a4c-ae47-599c732a9635 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:25.874750 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b04a796-0b1d-4a4c-ae47-599c732a9635 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:25.893491 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-6b04a796-0b1d-4a4c-ae47-599c732a9635 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:06:25.894690 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-6b04a796-0b1d-4a4c-ae47-599c732a9635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:25.953391 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1292/2555] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:06:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:06:25.964026 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b25296f-8828-4192-97be-84092290ff97 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:25.965450 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-5b25296f-8828-4192-97be-84092290ff97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:26.023356 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b25296f-8828-4192-97be-84092290ff97 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:26.027070 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968309f2-8429-4ead-95fd-6fc08c86a48e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:26.027708 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968309f2-8429-4ead-95fd-6fc08c86a48e service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:26.027995 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968309f2-8429-4ead-95fd-6fc08c86a48e service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:26.028271 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968309f2-8429-4ead-95fd-6fc08c86a48e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:26.037352 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968309f2-8429-4ead-95fd-6fc08c86a48e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:26.039489 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968309f2-8429-4ead-95fd-6fc08c86a48e service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:26.048928 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1264/2556] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:06:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:06:26.322781 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b04a796-0b1d-4a4c-ae47-599c732a9635 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:26.324415 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-6b04a796-0b1d-4a4c-ae47-599c732a9635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:26.340889 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b04a796-0b1d-4a4c-ae47-599c732a9635 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:26.347044 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5219c2-047a-4be2-88b5-78fe04323297 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:26.347532 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5219c2-047a-4be2-88b5-78fe04323297 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:26.347861 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5219c2-047a-4be2-88b5-78fe04323297 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:26.348141 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5219c2-047a-4be2-88b5-78fe04323297 admin admin] PATH_INFO: `/v3/projects/9fe5bc9b3d0b423dab3a2d6aba289afa` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:26.349760 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5219c2-047a-4be2-88b5-78fe04323297 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9fe5bc9b3d0b423dab3a2d6aba289afa)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:26.352223 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5219c2-047a-4be2-88b5-78fe04323297 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:26.398514 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-fb5219c2-047a-4be2-88b5-78fe04323297 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9fe5bc9b3d0b423dab3a2d6aba289afa', 'request_id': 'req-fb5219c2-047a-4be2-88b5-78fe04323297'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:26.404057 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-fb5219c2-047a-4be2-88b5-78fe04323297 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9fe5bc9b3d0b423dab3a2d6aba289afa', 'request_id': 'req-fb5219c2-047a-4be2-88b5-78fe04323297'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:26.410062 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-fb5219c2-047a-4be2-88b5-78fe04323297 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9fe5bc9b3d0b423dab3a2d6aba289afa', 'request_id': 'req-fb5219c2-047a-4be2-88b5-78fe04323297'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:26.422952 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-fb5219c2-047a-4be2-88b5-78fe04323297 admin admin] The token cache is being invalidate because project 9fe5bc9b3d0b423dab3a2d6aba289afa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:06:26.423240 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-fb5219c2-047a-4be2-88b5-78fe04323297 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:26.423867 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-fb5219c2-047a-4be2-88b5-78fe04323297 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:06:26.425352 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1293/2557] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:06:26 2022] DELETE /identity/v3/projects/9fe5bc9b3d0b423dab3a2d6aba289afa => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:06:26.430550 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b308f65c-7864-4854-bb62-44b2657c5fa9 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:26.430834 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b308f65c-7864-4854-bb62-44b2657c5fa9 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:26.431106 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b308f65c-7864-4854-bb62-44b2657c5fa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:26.451450 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-b308f65c-7864-4854-bb62-44b2657c5fa9 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:06:26.452680 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-b308f65c-7864-4854-bb62-44b2657c5fa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:26.510908 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1265/2558] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:06:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:06:26.520116 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c57143-ff16-43d0-95b1-801066ae1fd0 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:26.520399 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c57143-ff16-43d0-95b1-801066ae1fd0 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:26.520672 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c57143-ff16-43d0-95b1-801066ae1fd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:26.540859 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-d9c57143-ff16-43d0-95b1-801066ae1fd0 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:06:26.542044 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d9c57143-ff16-43d0-95b1-801066ae1fd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:26.549759 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1294/2559] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:06:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:06:26.554059 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b308f65c-7864-4854-bb62-44b2657c5fa9 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:26.563980 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b308f65c-7864-4854-bb62-44b2657c5fa9 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:26.569440 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb22a30-4968-4ea5-91d9-05a0ad6402b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:26.569978 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb22a30-4968-4ea5-91d9-05a0ad6402b7 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:26.570263 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb22a30-4968-4ea5-91d9-05a0ad6402b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:26.570558 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb22a30-4968-4ea5-91d9-05a0ad6402b7 admin admin] PATH_INFO: `/v3/domains` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:26.571185 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb22a30-4968-4ea5-91d9-05a0ad6402b7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=96564) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 31 15:06:26.571491 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb22a30-4968-4ea5-91d9-05a0ad6402b7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:26.573112 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb22a30-4968-4ea5-91d9-05a0ad6402b7 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:26.585161 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1266/2560] 10.222.0.36 () {58 vars in 1153 bytes} [Mon Oct 31 15:06:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 31 15:06:26.589503 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c57143-ff16-43d0-95b1-801066ae1fd0 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:26.598794 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c57143-ff16-43d0-95b1-801066ae1fd0 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:26.603476 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad782dc-ecb3-4683-8aa3-b927318b2e79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:26.604100 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad782dc-ecb3-4683-8aa3-b927318b2e79 admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:26.604379 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad782dc-ecb3-4683-8aa3-b927318b2e79 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:26.604651 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad782dc-ecb3-4683-8aa3-b927318b2e79 admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:26.605290 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad782dc-ecb3-4683-8aa3-b927318b2e79 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:26.607193 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad782dc-ecb3-4683-8aa3-b927318b2e79 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:26.633149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:06:26.645339 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1295/2561] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:06:26 2022] POST /identity/v3/projects => generated 385 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:06:26.650562 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb22a30-4968-4ea5-91d9-05a0ad6402b7 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:26.660407 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb22a30-4968-4ea5-91d9-05a0ad6402b7 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:26.681122 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:06:26.719580 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba55443-a1bf-41bb-90d8-631d24b77fc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:26.720200 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba55443-a1bf-41bb-90d8-631d24b77fc3 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:26.720738 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba55443-a1bf-41bb-90d8-631d24b77fc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:26.720738 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba55443-a1bf-41bb-90d8-631d24b77fc3 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:26.721358 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba55443-a1bf-41bb-90d8-631d24b77fc3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:26.723272 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba55443-a1bf-41bb-90d8-631d24b77fc3 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:26.748473 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1267/2562] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:06:26 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:06:26.749699 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad782dc-ecb3-4683-8aa3-b927318b2e79 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:26.760074 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad782dc-ecb3-4683-8aa3-b927318b2e79 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:26.764603 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e59ccbf-c41c-404f-a344-48e67962128f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:26.765193 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e59ccbf-c41c-404f-a344-48e67962128f admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:26.765493 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e59ccbf-c41c-404f-a344-48e67962128f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:26.765764 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e59ccbf-c41c-404f-a344-48e67962128f admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:26.766395 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e59ccbf-c41c-404f-a344-48e67962128f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:26.768043 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e59ccbf-c41c-404f-a344-48e67962128f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:26.783383 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1296/2563] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:06:26 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:06:26.787267 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba55443-a1bf-41bb-90d8-631d24b77fc3 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:26.797918 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba55443-a1bf-41bb-90d8-631d24b77fc3 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:26.802775 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4212a46-f03c-4a4d-abc8-9200c3a985c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:26.804086 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4212a46-f03c-4a4d-abc8-9200c3a985c7 admin admin] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:26.804372 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4212a46-f03c-4a4d-abc8-9200c3a985c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:26.804667 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4212a46-f03c-4a4d-abc8-9200c3a985c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:26.805287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4212a46-f03c-4a4d-abc8-9200c3a985c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:26.806891 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4212a46-f03c-4a4d-abc8-9200c3a985c7 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:26.816935 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1268/2564] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:06:26 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:06:26.822097 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e59ccbf-c41c-404f-a344-48e67962128f admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:26.831905 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e59ccbf-c41c-404f-a344-48e67962128f admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:26.837293 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e55963e-6a7a-42fd-b8a3-f8c75b127d25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:26.837935 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e55963e-6a7a-42fd-b8a3-f8c75b127d25 admin admin] REQUEST_METHOD: `PUT` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:26.838216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e55963e-6a7a-42fd-b8a3-f8c75b127d25 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:26.838491 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e55963e-6a7a-42fd-b8a3-f8c75b127d25 admin admin] PATH_INFO: `/v3/projects/87556dde63c246f78107d7191fe9a177/users/0ba4c7a65d8b41e3addbbeba841d79ea/roles/bb354a7b796d467d91ff4b184011796e` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:26.853013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e55963e-6a7a-42fd-b8a3-f8c75b127d25 admin admin] RBAC: Authorizing `identity:create_grant(project_id=87556dde63c246f78107d7191fe9a177, user_id=0ba4c7a65d8b41e3addbbeba841d79ea, role_id=bb354a7b796d467d91ff4b184011796e)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:26.855680 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e55963e-6a7a-42fd-b8a3-f8c75b127d25 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:26.869149 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1297/2565] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:06:26 2022] PUT /identity/v3/projects/87556dde63c246f78107d7191fe9a177/users/0ba4c7a65d8b41e3addbbeba841d79ea/roles/bb354a7b796d467d91ff4b184011796e => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:06:26.874093 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaf2234-4fac-458b-989e-328b749d2083 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:26.874379 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaf2234-4fac-458b-989e-328b749d2083 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:26.874657 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaf2234-4fac-458b-989e-328b749d2083 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:26.896772 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-cbaf2234-4fac-458b-989e-328b749d2083 None None] MFA Rules not processed for user `7c12195059ec40bfafbb7be8e9f46734`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:06:26.898582 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-cbaf2234-4fac-458b-989e-328b749d2083 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:26.958685 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1269/2566] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:06:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:06:26.968533 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e55963e-6a7a-42fd-b8a3-f8c75b127d25 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:26.970353 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0e55963e-6a7a-42fd-b8a3-f8c75b127d25 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:27.026496 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e55963e-6a7a-42fd-b8a3-f8c75b127d25 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:27.030166 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6aea90-89e3-4e2d-92cc-9ee313f47a9a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:27.030768 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6aea90-89e3-4e2d-92cc-9ee313f47a9a service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:27.031049 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6aea90-89e3-4e2d-92cc-9ee313f47a9a service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:27.031320 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6aea90-89e3-4e2d-92cc-9ee313f47a9a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:27.040454 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6aea90-89e3-4e2d-92cc-9ee313f47a9a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:27.041865 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6aea90-89e3-4e2d-92cc-9ee313f47a9a service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:27.051143 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1298/2567] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:06:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:06:30.974465 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a016daa1-9390-448a-a6ae-4b12d09606b4 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:30.975036 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a016daa1-9390-448a-a6ae-4b12d09606b4 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:30.975776 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a016daa1-9390-448a-a6ae-4b12d09606b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:31.005442 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-a016daa1-9390-448a-a6ae-4b12d09606b4 None None] MFA Rules not processed for user `0ba4c7a65d8b41e3addbbeba841d79ea`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:06:31.007763 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a016daa1-9390-448a-a6ae-4b12d09606b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:31.082256 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1270/2568] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:06:30 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:06:31.087332 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6aea90-89e3-4e2d-92cc-9ee313f47a9a service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:31.097622 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6aea90-89e3-4e2d-92cc-9ee313f47a9a service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:31.102380 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90062d2e-11b1-490e-acdf-3ffa90a39f2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:31.102978 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90062d2e-11b1-490e-acdf-3ffa90a39f2f admin admin] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:31.103263 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90062d2e-11b1-490e-acdf-3ffa90a39f2f admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:31.103531 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90062d2e-11b1-490e-acdf-3ffa90a39f2f admin admin] PATH_INFO: `/v3/projects` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:31.104225 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90062d2e-11b1-490e-acdf-3ffa90a39f2f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:31.106275 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90062d2e-11b1-490e-acdf-3ffa90a39f2f admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:31.138361 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1299/2569] 10.222.0.36 () {60 vars in 1153 bytes} [Mon Oct 31 15:06:31 2022] POST /identity/v3/projects => generated 383 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 31 15:06:31.139448 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a016daa1-9390-448a-a6ae-4b12d09606b4 None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:31.151602 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a016daa1-9390-448a-a6ae-4b12d09606b4 None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:31.182253 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:06:31.211803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afec5622-fc08-4e22-8df8-a1ddee598811 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:31.212418 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afec5622-fc08-4e22-8df8-a1ddee598811 admin admin] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:31.212700 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afec5622-fc08-4e22-8df8-a1ddee598811 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:31.213044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afec5622-fc08-4e22-8df8-a1ddee598811 admin admin] PATH_INFO: `/v3/users` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:31.213677 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afec5622-fc08-4e22-8df8-a1ddee598811 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:31.215609 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afec5622-fc08-4e22-8df8-a1ddee598811 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:31.236187 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1271/2570] 10.222.0.36 () {60 vars in 1144 bytes} [Mon Oct 31 15:06:31 2022] POST /identity/v3/users => generated 298 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 31 15:06:31.240328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90062d2e-11b1-490e-acdf-3ffa90a39f2f admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:31.251163 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90062d2e-11b1-490e-acdf-3ffa90a39f2f admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:31.256073 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9ba668-9dce-4fb1-86cf-a80b24a6eccf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:31.257355 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9ba668-9dce-4fb1-86cf-a80b24a6eccf admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:31.257501 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9ba668-9dce-4fb1-86cf-a80b24a6eccf admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:31.257757 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9ba668-9dce-4fb1-86cf-a80b24a6eccf admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:31.258386 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9ba668-9dce-4fb1-86cf-a80b24a6eccf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:31.260047 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9ba668-9dce-4fb1-86cf-a80b24a6eccf admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:31.270669 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1300/2571] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:06:31 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:06:31.275723 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afec5622-fc08-4e22-8df8-a1ddee598811 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:31.285617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afec5622-fc08-4e22-8df8-a1ddee598811 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:31.290332 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5716f936-b63a-4eaa-bb12-c07317691d68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:31.290969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5716f936-b63a-4eaa-bb12-c07317691d68 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:31.291245 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5716f936-b63a-4eaa-bb12-c07317691d68 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:31.291517 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5716f936-b63a-4eaa-bb12-c07317691d68 admin admin] PATH_INFO: `/v3/projects/36697cb548a14f47a84f7eebd452983f/users/fd187260f23542978e2d95520bd8f0c1/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:31.306461 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5716f936-b63a-4eaa-bb12-c07317691d68 admin admin] RBAC: Authorizing `identity:create_grant(project_id=36697cb548a14f47a84f7eebd452983f, user_id=fd187260f23542978e2d95520bd8f0c1, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:31.309121 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5716f936-b63a-4eaa-bb12-c07317691d68 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:31.323040 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1272/2572] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:06:31 2022] PUT /identity/v3/projects/36697cb548a14f47a84f7eebd452983f/users/fd187260f23542978e2d95520bd8f0c1/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:06:31.327012 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9ba668-9dce-4fb1-86cf-a80b24a6eccf admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:31.338219 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9ba668-9dce-4fb1-86cf-a80b24a6eccf admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:31.397813 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be582d8-670b-44d9-9fcd-4e9e34233875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:31.398421 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be582d8-670b-44d9-9fcd-4e9e34233875 admin admin] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:31.398706 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be582d8-670b-44d9-9fcd-4e9e34233875 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:31.398984 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be582d8-670b-44d9-9fcd-4e9e34233875 admin admin] PATH_INFO: `/v3/roles` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:31.399656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be582d8-670b-44d9-9fcd-4e9e34233875 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:31.401269 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be582d8-670b-44d9-9fcd-4e9e34233875 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:31.410860 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1301/2573] 10.222.0.36 () {58 vars in 1122 bytes} [Mon Oct 31 15:06:31 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 31 15:06:31.415047 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5716f936-b63a-4eaa-bb12-c07317691d68 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:31.425333 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5716f936-b63a-4eaa-bb12-c07317691d68 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:31.430653 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50dca39-cc65-4126-b490-066122c1e472 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:31.431288 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50dca39-cc65-4126-b490-066122c1e472 admin admin] REQUEST_METHOD: `PUT` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:31.431599 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50dca39-cc65-4126-b490-066122c1e472 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:31.431875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50dca39-cc65-4126-b490-066122c1e472 admin admin] PATH_INFO: `/v3/projects/36697cb548a14f47a84f7eebd452983f/users/fd187260f23542978e2d95520bd8f0c1/roles/57572a37675d4c418e13794a7b8f3a5d` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:31.433769 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50dca39-cc65-4126-b490-066122c1e472 admin admin] RBAC: Authorizing `identity:create_grant(project_id=36697cb548a14f47a84f7eebd452983f, user_id=fd187260f23542978e2d95520bd8f0c1, role_id=57572a37675d4c418e13794a7b8f3a5d)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:31.436413 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50dca39-cc65-4126-b490-066122c1e472 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:31.447657 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1273/2574] 10.222.0.36 () {60 vars in 1483 bytes} [Mon Oct 31 15:06:31 2022] PUT /identity/v3/projects/36697cb548a14f47a84f7eebd452983f/users/fd187260f23542978e2d95520bd8f0c1/roles/57572a37675d4c418e13794a7b8f3a5d => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:06:36.030663 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4ce198-5991-4c0d-8733-c73056e2510b None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:36.031504 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4ce198-5991-4c0d-8733-c73056e2510b None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:36.032058 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4ce198-5991-4c0d-8733-c73056e2510b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:36.063591 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-1c4ce198-5991-4c0d-8733-c73056e2510b None None] MFA Rules not processed for user `fd187260f23542978e2d95520bd8f0c1`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:06:36.064856 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1c4ce198-5991-4c0d-8733-c73056e2510b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:36.081648 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:06:36.148147 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1302/2575] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:06:36 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:06:36.164758 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50dca39-cc65-4126-b490-066122c1e472 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:36.167254 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-f50dca39-cc65-4126-b490-066122c1e472 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:36.232464 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50dca39-cc65-4126-b490-066122c1e472 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:36.236234 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407d280e-2e4a-48f9-8f89-ee6841e4c0fe service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9808dcba0b64027bb70e31d1569957d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:36.236851 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407d280e-2e4a-48f9-8f89-ee6841e4c0fe service glance] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:36.237193 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407d280e-2e4a-48f9-8f89-ee6841e4c0fe service glance] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:36.237400 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407d280e-2e4a-48f9-8f89-ee6841e4c0fe service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:36.306911 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407d280e-2e4a-48f9-8f89-ee6841e4c0fe service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:36.308390 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407d280e-2e4a-48f9-8f89-ee6841e4c0fe service glance] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:36.319618 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1274/2576] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:06:36 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:06:36.464068 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2415236-3d89-4b1d-bf7b-587a1acf9191 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:36.464704 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2415236-3d89-4b1d-bf7b-587a1acf9191 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:36.465582 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2415236-3d89-4b1d-bf7b-587a1acf9191 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:36.498091 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-d2415236-3d89-4b1d-bf7b-587a1acf9191 None None] MFA Rules not processed for user `e0d0f8598fac40449bcb2a55ff8df8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:06:36.499452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-d2415236-3d89-4b1d-bf7b-587a1acf9191 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:36.557702 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1303/2577] 10.222.0.36 () {58 vars in 1009 bytes} [Mon Oct 31 15:06:36 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:06:36.604634 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407d280e-2e4a-48f9-8f89-ee6841e4c0fe service glance] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:36.606054 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-407d280e-2e4a-48f9-8f89-ee6841e4c0fe service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:36.649832 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:06:36.678838 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407d280e-2e4a-48f9-8f89-ee6841e4c0fe service glance] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:36.683907 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83773039-6cad-41f1-ac95-02db677caa24 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4ae5e20cc5b40ccbdb951f908f5080d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:36.684511 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83773039-6cad-41f1-ac95-02db677caa24 service swift] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:36.684791 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83773039-6cad-41f1-ac95-02db677caa24 service swift] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:36.685064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83773039-6cad-41f1-ac95-02db677caa24 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:36.693285 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83773039-6cad-41f1-ac95-02db677caa24 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:36.694892 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83773039-6cad-41f1-ac95-02db677caa24 service swift] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:36.702880 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1275/2578] 10.222.0.36 () {60 vars in 1384 bytes} [Mon Oct 31 15:06:36 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 31 15:06:43.271319 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2415236-3d89-4b1d-bf7b-587a1acf9191 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:43.341465 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2415236-3d89-4b1d-bf7b-587a1acf9191 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:43.347209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4dcd3c-69d8-423b-bc47-84f32974aa71 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:43.347209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4dcd3c-69d8-423b-bc47-84f32974aa71 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:43.347209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4dcd3c-69d8-423b-bc47-84f32974aa71 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:43.347209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4dcd3c-69d8-423b-bc47-84f32974aa71 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:43.348511 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-9f4dcd3c-69d8-423b-bc47-84f32974aa71 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:43.419773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=100 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:06:43.438573 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4dcd3c-69d8-423b-bc47-84f32974aa71 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:43.440455 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4dcd3c-69d8-423b-bc47-84f32974aa71 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:43.454809 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1304/2579] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:06:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:06:44.654648 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83773039-6cad-41f1-ac95-02db677caa24 service swift] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:44.656322 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-83773039-6cad-41f1-ac95-02db677caa24 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:44.670075 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83773039-6cad-41f1-ac95-02db677caa24 service swift] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:44.673895 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d7b6ac-f798-47b9-8c0c-ed56d76cd495 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:44.674528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d7b6ac-f798-47b9-8c0c-ed56d76cd495 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:44.674808 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d7b6ac-f798-47b9-8c0c-ed56d76cd495 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:44.675079 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d7b6ac-f798-47b9-8c0c-ed56d76cd495 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:44.676975 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-b4d7b6ac-f798-47b9-8c0c-ed56d76cd495 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:44.693504 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d7b6ac-f798-47b9-8c0c-ed56d76cd495 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:44.695803 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d7b6ac-f798-47b9-8c0c-ed56d76cd495 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:44.708729 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1276/2580] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:06:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:06:46.429281 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03a96fb-4d0f-4197-95b3-67c3aed563dc None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:46.429784 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03a96fb-4d0f-4197-95b3-67c3aed563dc None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:46.429889 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03a96fb-4d0f-4197-95b3-67c3aed563dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:46.449767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-a03a96fb-4d0f-4197-95b3-67c3aed563dc None None] MFA Rules not processed for user `0ba4c7a65d8b41e3addbbeba841d79ea`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:06:46.450962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-a03a96fb-4d0f-4197-95b3-67c3aed563dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:46.457824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1305/2581] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:06:46 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:06:50.027113 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43252306-46fe-4065-99a3-7f85a45a8fe6 None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:50.027641 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43252306-46fe-4065-99a3-7f85a45a8fe6 None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:50.027751 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43252306-46fe-4065-99a3-7f85a45a8fe6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:50.080065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-43252306-46fe-4065-99a3-7f85a45a8fe6 None None] MFA Rules not processed for user `e0d0f8598fac40449bcb2a55ff8df8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:06:50.081302 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-43252306-46fe-4065-99a3-7f85a45a8fe6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:06:50.089521 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1277/2582] 10.222.0.36 () {58 vars in 1009 bytes} [Mon Oct 31 15:06:50 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:06:50.102103 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03a96fb-4d0f-4197-95b3-67c3aed563dc None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:06:50.114372 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03a96fb-4d0f-4197-95b3-67c3aed563dc None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:06:50.121022 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bdfb82-df4b-4041-9f51-1f0f9e711933 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4ae5e20cc5b40ccbdb951f908f5080d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:06:50.121675 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bdfb82-df4b-4041-9f51-1f0f9e711933 service swift] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:06:50.121962 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bdfb82-df4b-4041-9f51-1f0f9e711933 service swift] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:06:50.122240 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bdfb82-df4b-4041-9f51-1f0f9e711933 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:06:50.131628 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bdfb82-df4b-4041-9f51-1f0f9e711933 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:06:50.133279 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bdfb82-df4b-4041-9f51-1f0f9e711933 service swift] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:06:50.141877 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1306/2583] 10.222.0.36 () {60 vars in 1384 bytes} [Mon Oct 31 15:06:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 31 15:07:00.077522 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=34 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:07:00.138409 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=14 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:07:00.420182 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c97e84-c154-4f72-8e1b-a605648a43ca None None] REQUEST_METHOD: `POST` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:07:00.420552 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c97e84-c154-4f72-8e1b-a605648a43ca None None] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:07:00.420930 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c97e84-c154-4f72-8e1b-a605648a43ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:07:00.445105 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.auth.core [None req-18c97e84-c154-4f72-8e1b-a605648a43ca None None] MFA Rules not processed for user `f0b358738b5a46d8bc1db1da68040c8f`. Rule list: `[]` (Enabled: `True`). {{(pid=96564) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:07:00.446640 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-18c97e84-c154-4f72-8e1b-a605648a43ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:07:00.513604 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1278/2584] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:07:00 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:07:04.220062 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bdfb82-df4b-4041-9f51-1f0f9e711933 service swift] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:07:04.230823 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bdfb82-df4b-4041-9f51-1f0f9e711933 service swift] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:07:04.234557 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632d7a94-2bce-4f28-a83f-d9666cd9fe84 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:07:04.235253 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632d7a94-2bce-4f28-a83f-d9666cd9fe84 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:07:04.235797 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632d7a94-2bce-4f28-a83f-d9666cd9fe84 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:07:04.236154 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632d7a94-2bce-4f28-a83f-d9666cd9fe84 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:07:04.237973 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-632d7a94-2bce-4f28-a83f-d9666cd9fe84 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:07:04.249539 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632d7a94-2bce-4f28-a83f-d9666cd9fe84 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:07:04.251240 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632d7a94-2bce-4f28-a83f-d9666cd9fe84 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:07:04.260373 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1307/2585] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:07:04 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:07:04.369303 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c97e84-c154-4f72-8e1b-a605648a43ca None None] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:07:04.378947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c97e84-c154-4f72-8e1b-a605648a43ca None None] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:07:04.384044 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb21a5d-85d7-41b2-a47b-1109bc3ea75f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:07:04.384674 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb21a5d-85d7-41b2-a47b-1109bc3ea75f service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:07:04.384945 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb21a5d-85d7-41b2-a47b-1109bc3ea75f service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:07:04.385431 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb21a5d-85d7-41b2-a47b-1109bc3ea75f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:07:04.387993 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-3bb21a5d-85d7-41b2-a47b-1109bc3ea75f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:07:04.407427 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb21a5d-85d7-41b2-a47b-1109bc3ea75f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:07:04.407427 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb21a5d-85d7-41b2-a47b-1109bc3ea75f service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:07:04.418598 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1279/2586] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:07:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:07:13.390200 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632d7a94-2bce-4f28-a83f-d9666cd9fe84 service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:07:13.399806 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632d7a94-2bce-4f28-a83f-d9666cd9fe84 service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:07:13.403537 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a062cff-0075-407a-a749-692526d3201e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:07:13.404141 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a062cff-0075-407a-a749-692526d3201e service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:07:13.404418 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a062cff-0075-407a-a749-692526d3201e service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:07:13.404688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a062cff-0075-407a-a749-692526d3201e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:07:13.406424 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-6a062cff-0075-407a-a749-692526d3201e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:07:13.417548 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a062cff-0075-407a-a749-692526d3201e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:07:13.418961 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a062cff-0075-407a-a749-692526d3201e service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:07:13.426952 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1308/2587] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:07:13 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:07:14.413561 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=31 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:07:18.033287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb21a5d-85d7-41b2-a47b-1109bc3ea75f service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:07:18.042645 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb21a5d-85d7-41b2-a47b-1109bc3ea75f service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:07:18.046984 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904d7662-8e34-46e0-8c05-ab28ceebb810 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:07:18.047622 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904d7662-8e34-46e0-8c05-ab28ceebb810 service neutron] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:07:18.047893 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904d7662-8e34-46e0-8c05-ab28ceebb810 service neutron] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:07:18.048166 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904d7662-8e34-46e0-8c05-ab28ceebb810 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:07:18.057523 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904d7662-8e34-46e0-8c05-ab28ceebb810 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:07:18.058943 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904d7662-8e34-46e0-8c05-ab28ceebb810 service neutron] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:07:18.070388 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1280/2588] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:07:18 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:07:19.993146 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a062cff-0075-407a-a749-692526d3201e service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:07:19.994722 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-6a062cff-0075-407a-a749-692526d3201e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:07:20.013730 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a062cff-0075-407a-a749-692526d3201e service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:07:20.019833 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09549251-aa40-4d4e-9027-36acf64cf333 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:07:20.021303 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09549251-aa40-4d4e-9027-36acf64cf333 service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:07:20.021720 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09549251-aa40-4d4e-9027-36acf64cf333 service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:07:20.022083 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09549251-aa40-4d4e-9027-36acf64cf333 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:07:20.024043 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-09549251-aa40-4d4e-9027-36acf64cf333 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:07:20.039303 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09549251-aa40-4d4e-9027-36acf64cf333 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:07:20.041011 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09549251-aa40-4d4e-9027-36acf64cf333 service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:07:20.053395 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1309/2589] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:07:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:07:28.065434 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=6 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:07:30.048105 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=18 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:08:36.403132 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904d7662-8e34-46e0-8c05-ab28ceebb810 service neutron] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:08:36.415588 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904d7662-8e34-46e0-8c05-ab28ceebb810 service neutron] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:08:36.421175 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2db45f-f8b8-4f8a-8b82-ad062d223da9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:08:36.421770 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2db45f-f8b8-4f8a-8b82-ad062d223da9 service nova] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:08:36.422053 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2db45f-f8b8-4f8a-8b82-ad062d223da9 service nova] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:08:36.422326 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2db45f-f8b8-4f8a-8b82-ad062d223da9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:08:36.424111 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ad2db45f-f8b8-4f8a-8b82-ad062d223da9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:36.483735 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2db45f-f8b8-4f8a-8b82-ad062d223da9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:08:36.485336 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2db45f-f8b8-4f8a-8b82-ad062d223da9 service nova] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:08:36.494489 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1281/2590] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:08:36 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:08:37.908165 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09549251-aa40-4d4e-9027-36acf64cf333 service nova] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:08:37.918534 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09549251-aa40-4d4e-9027-36acf64cf333 service nova] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:08:37.922796 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734bcd90-ddae-4792-857a-d88c2bc309cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:08:37.923381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734bcd90-ddae-4792-857a-d88c2bc309cf service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:08:37.923766 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734bcd90-ddae-4792-857a-d88c2bc309cf service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:08:37.924145 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734bcd90-ddae-4792-857a-d88c2bc309cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:08:37.925839 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-734bcd90-ddae-4792-857a-d88c2bc309cf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:37.992256 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734bcd90-ddae-4792-857a-d88c2bc309cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:08:37.994063 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734bcd90-ddae-4792-857a-d88c2bc309cf service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:08:38.003742 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1310/2591] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:08:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:08:41.849202 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2db45f-f8b8-4f8a-8b82-ad062d223da9 service nova] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:08:41.851483 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-ad2db45f-f8b8-4f8a-8b82-ad062d223da9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:41.948311 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2db45f-f8b8-4f8a-8b82-ad062d223da9 service nova] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:08:41.953919 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415cc396-133b-4610-811c-c3cd4a714a9e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70ea2aaf529b46dabb70b71f4dcc3860', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:08:41.954875 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415cc396-133b-4610-811c-c3cd4a714a9e service placement] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:08:41.955224 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415cc396-133b-4610-811c-c3cd4a714a9e service placement] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:08:41.955610 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415cc396-133b-4610-811c-c3cd4a714a9e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:08:41.957428 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-415cc396-133b-4610-811c-c3cd4a714a9e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:41.971126 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415cc396-133b-4610-811c-c3cd4a714a9e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:08:41.972801 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415cc396-133b-4610-811c-c3cd4a714a9e service placement] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:08:41.986617 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1282/2592] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:08:41 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:08:42.062307 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c59991c-1bd6-4bf1-a713-fceeb3b71482 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:08:42.062792 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c59991c-1bd6-4bf1-a713-fceeb3b71482 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:08:42.062907 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c59991c-1bd6-4bf1-a713-fceeb3b71482 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:08:42.083227 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-1c59991c-1bd6-4bf1-a713-fceeb3b71482 None None] MFA Rules not processed for user `e0d0f8598fac40449bcb2a55ff8df8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:08:42.084469 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1c59991c-1bd6-4bf1-a713-fceeb3b71482 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:42.102356 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1311/2593] 10.222.0.36 () {58 vars in 967 bytes} [Mon Oct 31 15:08:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:08:42.142599 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415cc396-133b-4610-811c-c3cd4a714a9e service placement] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:08:42.163897 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415cc396-133b-4610-811c-c3cd4a714a9e service placement] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:08:42.169321 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a857776b-4d78-407a-8c14-a3fd083d092d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4ae5e20cc5b40ccbdb951f908f5080d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:08:42.169921 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a857776b-4d78-407a-8c14-a3fd083d092d service swift] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:08:42.170203 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a857776b-4d78-407a-8c14-a3fd083d092d service swift] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:08:42.170469 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a857776b-4d78-407a-8c14-a3fd083d092d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:08:42.179360 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a857776b-4d78-407a-8c14-a3fd083d092d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:08:42.180981 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a857776b-4d78-407a-8c14-a3fd083d092d service swift] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:08:42.189240 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1283/2594] 10.222.0.36 () {60 vars in 1384 bytes} [Mon Oct 31 15:08:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 31 15:08:45.272094 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c59991c-1bd6-4bf1-a713-fceeb3b71482 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:08:45.288777 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c59991c-1bd6-4bf1-a713-fceeb3b71482 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:08:45.298616 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0756c57d-25f6-4751-b60b-093382851f89 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70ea2aaf529b46dabb70b71f4dcc3860', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:08:45.301244 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a857776b-4d78-407a-8c14-a3fd083d092d service swift] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:08:45.302717 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-a857776b-4d78-407a-8c14-a3fd083d092d service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:45.304376 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0756c57d-25f6-4751-b60b-093382851f89 service placement] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:08:45.305014 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0756c57d-25f6-4751-b60b-093382851f89 service placement] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:08:45.305449 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0756c57d-25f6-4751-b60b-093382851f89 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:08:45.317731 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0756c57d-25f6-4751-b60b-093382851f89 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:45.334008 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0756c57d-25f6-4751-b60b-093382851f89 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:08:45.337078 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0756c57d-25f6-4751-b60b-093382851f89 service placement] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:08:45.340070 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a857776b-4d78-407a-8c14-a3fd083d092d service swift] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:08:45.346187 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf2dc08-7a59-4a7b-8ce8-f5824764c38f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:08:45.347259 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf2dc08-7a59-4a7b-8ce8-f5824764c38f admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:08:45.347942 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf2dc08-7a59-4a7b-8ce8-f5824764c38f admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:08:45.348989 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf2dc08-7a59-4a7b-8ce8-f5824764c38f admin admin] PATH_INFO: `/v3/users/c6dc53b3c9e041a7ae977b8f9c4fb478` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:08:45.350401 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf2dc08-7a59-4a7b-8ce8-f5824764c38f admin admin] RBAC: Authorizing `identity:delete_user(user_id=c6dc53b3c9e041a7ae977b8f9c4fb478)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:08:45.352986 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf2dc08-7a59-4a7b-8ce8-f5824764c38f admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:08:45.354452 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1312/2595] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:08:45 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:08:45.419807 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-faf2dc08-7a59-4a7b-8ce8-f5824764c38f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c6dc53b3c9e041a7ae977b8f9c4fb478'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:45.428282 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-faf2dc08-7a59-4a7b-8ce8-f5824764c38f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c6dc53b3c9e041a7ae977b8f9c4fb478'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:45.429213 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-faf2dc08-7a59-4a7b-8ce8-f5824764c38f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c6dc53b3c9e041a7ae977b8f9c4fb478'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:45.430061 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-faf2dc08-7a59-4a7b-8ce8-f5824764c38f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c6dc53b3c9e041a7ae977b8f9c4fb478'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:45.436064 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:08:45.449374 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-faf2dc08-7a59-4a7b-8ce8-f5824764c38f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c6dc53b3c9e041a7ae977b8f9c4fb478'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:45.465341 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1284/2596] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:08:45 2022] DELETE /identity/v3/users/c6dc53b3c9e041a7ae977b8f9c4fb478 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 31 15:08:45.723735 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0756c57d-25f6-4751-b60b-093382851f89 service placement] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:08:45.736674 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0756c57d-25f6-4751-b60b-093382851f89 service placement] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:08:45.873001 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43654f52-b022-4cb6-b6a4-9e66dc48086a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:08:45.873649 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43654f52-b022-4cb6-b6a4-9e66dc48086a admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:08:45.873929 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43654f52-b022-4cb6-b6a4-9e66dc48086a admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:08:45.874204 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43654f52-b022-4cb6-b6a4-9e66dc48086a admin admin] PATH_INFO: `/v3/projects/357cf607c19548859a14c92a4e90f148` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:08:45.875397 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43654f52-b022-4cb6-b6a4-9e66dc48086a admin admin] RBAC: Authorizing `identity:delete_project(project_id=357cf607c19548859a14c92a4e90f148)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:08:45.883824 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43654f52-b022-4cb6-b6a4-9e66dc48086a admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:08:45.958611 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-43654f52-b022-4cb6-b6a4-9e66dc48086a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '357cf607c19548859a14c92a4e90f148', 'request_id': 'req-43654f52-b022-4cb6-b6a4-9e66dc48086a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:45.969550 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-43654f52-b022-4cb6-b6a4-9e66dc48086a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '357cf607c19548859a14c92a4e90f148', 'request_id': 'req-43654f52-b022-4cb6-b6a4-9e66dc48086a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:45.978231 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-43654f52-b022-4cb6-b6a4-9e66dc48086a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '357cf607c19548859a14c92a4e90f148', 'request_id': 'req-43654f52-b022-4cb6-b6a4-9e66dc48086a'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:45.985351 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:08:45.999999 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-43654f52-b022-4cb6-b6a4-9e66dc48086a admin admin] The token cache is being invalidate because project 357cf607c19548859a14c92a4e90f148 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:08:46.000541 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-43654f52-b022-4cb6-b6a4-9e66dc48086a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:46.002139 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-43654f52-b022-4cb6-b6a4-9e66dc48086a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:46.007817 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1313/2597] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:08:45 2022] DELETE /identity/v3/projects/357cf607c19548859a14c92a4e90f148 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:08:46.010720 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf2dc08-7a59-4a7b-8ce8-f5824764c38f admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:08:46.012529 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-faf2dc08-7a59-4a7b-8ce8-f5824764c38f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:46.028627 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf2dc08-7a59-4a7b-8ce8-f5824764c38f admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:08:46.125059 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff8e01b-aae0-43e2-b443-a1c8b7b129ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:08:46.125721 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff8e01b-aae0-43e2-b443-a1c8b7b129ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:08:46.126005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff8e01b-aae0-43e2-b443-a1c8b7b129ad admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:08:46.126283 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff8e01b-aae0-43e2-b443-a1c8b7b129ad admin admin] PATH_INFO: `/v3/users/7b9180028dd048179e4247fe2e64a83b` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:08:46.127774 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff8e01b-aae0-43e2-b443-a1c8b7b129ad admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b9180028dd048179e4247fe2e64a83b)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:08:46.129937 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff8e01b-aae0-43e2-b443-a1c8b7b129ad admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:08:46.192686 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-eff8e01b-aae0-43e2-b443-a1c8b7b129ad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b9180028dd048179e4247fe2e64a83b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:46.201361 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-eff8e01b-aae0-43e2-b443-a1c8b7b129ad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b9180028dd048179e4247fe2e64a83b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:46.202169 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-eff8e01b-aae0-43e2-b443-a1c8b7b129ad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b9180028dd048179e4247fe2e64a83b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:46.203189 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-eff8e01b-aae0-43e2-b443-a1c8b7b129ad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b9180028dd048179e4247fe2e64a83b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:46.221136 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-eff8e01b-aae0-43e2-b443-a1c8b7b129ad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b9180028dd048179e4247fe2e64a83b'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:46.238435 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1285/2598] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:08:46 2022] DELETE /identity/v3/users/7b9180028dd048179e4247fe2e64a83b => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:08:46.499536 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43654f52-b022-4cb6-b6a4-9e66dc48086a admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:08:46.501344 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-43654f52-b022-4cb6-b6a4-9e66dc48086a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:46.517686 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43654f52-b022-4cb6-b6a4-9e66dc48086a admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:08:46.591618 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c804557-4c65-4ae8-a22f-ce340d29d152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:08:46.592263 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c804557-4c65-4ae8-a22f-ce340d29d152 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:08:46.592563 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c804557-4c65-4ae8-a22f-ce340d29d152 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:08:46.592817 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c804557-4c65-4ae8-a22f-ce340d29d152 admin admin] PATH_INFO: `/v3/projects/eb900e1a5dd74001934b483145f18364` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:08:46.593991 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c804557-4c65-4ae8-a22f-ce340d29d152 admin admin] RBAC: Authorizing `identity:delete_project(project_id=eb900e1a5dd74001934b483145f18364)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:08:46.596168 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c804557-4c65-4ae8-a22f-ce340d29d152 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:08:46.645688 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0c804557-4c65-4ae8-a22f-ce340d29d152 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb900e1a5dd74001934b483145f18364', 'request_id': 'req-0c804557-4c65-4ae8-a22f-ce340d29d152'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:46.652060 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0c804557-4c65-4ae8-a22f-ce340d29d152 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb900e1a5dd74001934b483145f18364', 'request_id': 'req-0c804557-4c65-4ae8-a22f-ce340d29d152'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:46.658651 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0c804557-4c65-4ae8-a22f-ce340d29d152 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb900e1a5dd74001934b483145f18364', 'request_id': 'req-0c804557-4c65-4ae8-a22f-ce340d29d152'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:46.672013 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0c804557-4c65-4ae8-a22f-ce340d29d152 admin admin] The token cache is being invalidate because project eb900e1a5dd74001934b483145f18364 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:08:46.672304 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0c804557-4c65-4ae8-a22f-ce340d29d152 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:46.672952 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-0c804557-4c65-4ae8-a22f-ce340d29d152 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:46.674511 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1314/2599] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:08:46 2022] DELETE /identity/v3/projects/eb900e1a5dd74001934b483145f18364 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:08:49.615758 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff8e01b-aae0-43e2-b443-a1c8b7b129ad admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:08:49.617361 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-eff8e01b-aae0-43e2-b443-a1c8b7b129ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:49.631255 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff8e01b-aae0-43e2-b443-a1c8b7b129ad admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:08:49.682289 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:08:49.689242 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f2908c-8cdd-496e-906e-85345c150551 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:08:49.690065 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f2908c-8cdd-496e-906e-85345c150551 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:08:49.690528 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f2908c-8cdd-496e-906e-85345c150551 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:08:49.691199 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f2908c-8cdd-496e-906e-85345c150551 admin admin] PATH_INFO: `/v3/users/0ba4c7a65d8b41e3addbbeba841d79ea` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:08:49.692557 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f2908c-8cdd-496e-906e-85345c150551 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0ba4c7a65d8b41e3addbbeba841d79ea)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:08:49.696272 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f2908c-8cdd-496e-906e-85345c150551 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:08:49.743005 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-e8f2908c-8cdd-496e-906e-85345c150551 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0ba4c7a65d8b41e3addbbeba841d79ea'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:49.750460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-e8f2908c-8cdd-496e-906e-85345c150551 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0ba4c7a65d8b41e3addbbeba841d79ea'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:49.751209 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-e8f2908c-8cdd-496e-906e-85345c150551 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0ba4c7a65d8b41e3addbbeba841d79ea'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:49.751903 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-e8f2908c-8cdd-496e-906e-85345c150551 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0ba4c7a65d8b41e3addbbeba841d79ea'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:49.766024 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-e8f2908c-8cdd-496e-906e-85345c150551 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0ba4c7a65d8b41e3addbbeba841d79ea'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:49.780442 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1286/2600] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:08:49 2022] DELETE /identity/v3/users/0ba4c7a65d8b41e3addbbeba841d79ea => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:08:49.952890 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c804557-4c65-4ae8-a22f-ce340d29d152 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:08:49.954464 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-0c804557-4c65-4ae8-a22f-ce340d29d152 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:49.968077 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c804557-4c65-4ae8-a22f-ce340d29d152 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:08:50.024784 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fc1661-9135-4d5e-89e8-24d2c1531d75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:08:50.025418 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fc1661-9135-4d5e-89e8-24d2c1531d75 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:08:50.025701 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fc1661-9135-4d5e-89e8-24d2c1531d75 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:08:50.026001 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fc1661-9135-4d5e-89e8-24d2c1531d75 admin admin] PATH_INFO: `/v3/projects/87556dde63c246f78107d7191fe9a177` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:08:50.027114 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fc1661-9135-4d5e-89e8-24d2c1531d75 admin admin] RBAC: Authorizing `identity:delete_project(project_id=87556dde63c246f78107d7191fe9a177)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:08:50.029050 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fc1661-9135-4d5e-89e8-24d2c1531d75 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:08:50.060648 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:08:50.078727 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-96fc1661-9135-4d5e-89e8-24d2c1531d75 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '87556dde63c246f78107d7191fe9a177', 'request_id': 'req-96fc1661-9135-4d5e-89e8-24d2c1531d75'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:50.084238 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-96fc1661-9135-4d5e-89e8-24d2c1531d75 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '87556dde63c246f78107d7191fe9a177', 'request_id': 'req-96fc1661-9135-4d5e-89e8-24d2c1531d75'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:50.090872 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-96fc1661-9135-4d5e-89e8-24d2c1531d75 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '87556dde63c246f78107d7191fe9a177', 'request_id': 'req-96fc1661-9135-4d5e-89e8-24d2c1531d75'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:50.102977 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-96fc1661-9135-4d5e-89e8-24d2c1531d75 admin admin] The token cache is being invalidate because project 87556dde63c246f78107d7191fe9a177 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:08:50.103266 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-96fc1661-9135-4d5e-89e8-24d2c1531d75 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:50.103939 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-96fc1661-9135-4d5e-89e8-24d2c1531d75 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:50.105444 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1315/2601] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:08:49 2022] DELETE /identity/v3/projects/87556dde63c246f78107d7191fe9a177 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:08:50.109421 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f2908c-8cdd-496e-906e-85345c150551 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:08:50.110843 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-e8f2908c-8cdd-496e-906e-85345c150551 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:50.124109 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f2908c-8cdd-496e-906e-85345c150551 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:08:50.180115 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215ff8e2-581b-4bbd-91d2-6a42c8272c11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:08:50.180744 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215ff8e2-581b-4bbd-91d2-6a42c8272c11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:08:50.181028 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215ff8e2-581b-4bbd-91d2-6a42c8272c11 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:08:50.181342 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215ff8e2-581b-4bbd-91d2-6a42c8272c11 admin admin] PATH_INFO: `/v3/users/fd187260f23542978e2d95520bd8f0c1` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:08:50.182461 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215ff8e2-581b-4bbd-91d2-6a42c8272c11 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fd187260f23542978e2d95520bd8f0c1)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:08:50.184460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215ff8e2-581b-4bbd-91d2-6a42c8272c11 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:08:50.231259 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-215ff8e2-581b-4bbd-91d2-6a42c8272c11 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fd187260f23542978e2d95520bd8f0c1'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:50.238097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-215ff8e2-581b-4bbd-91d2-6a42c8272c11 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fd187260f23542978e2d95520bd8f0c1'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:50.238746 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-215ff8e2-581b-4bbd-91d2-6a42c8272c11 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fd187260f23542978e2d95520bd8f0c1'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:50.239339 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-215ff8e2-581b-4bbd-91d2-6a42c8272c11 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fd187260f23542978e2d95520bd8f0c1'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:50.251606 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-215ff8e2-581b-4bbd-91d2-6a42c8272c11 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fd187260f23542978e2d95520bd8f0c1'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:50.255130 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:08:50.267014 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1287/2602] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:08:50 2022] DELETE /identity/v3/users/fd187260f23542978e2d95520bd8f0c1 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:08:50.437306 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fc1661-9135-4d5e-89e8-24d2c1531d75 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:08:50.438755 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-96fc1661-9135-4d5e-89e8-24d2c1531d75 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:50.454087 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fc1661-9135-4d5e-89e8-24d2c1531d75 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:08:50.517033 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38aa3633-c3ea-44d8-bb0f-d6ccac6c7951 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:08:50.517658 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38aa3633-c3ea-44d8-bb0f-d6ccac6c7951 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:08:50.517936 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38aa3633-c3ea-44d8-bb0f-d6ccac6c7951 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:08:50.518209 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38aa3633-c3ea-44d8-bb0f-d6ccac6c7951 admin admin] PATH_INFO: `/v3/projects/36697cb548a14f47a84f7eebd452983f` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:08:50.519337 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38aa3633-c3ea-44d8-bb0f-d6ccac6c7951 admin admin] RBAC: Authorizing `identity:delete_project(project_id=36697cb548a14f47a84f7eebd452983f)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:08:50.521256 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38aa3633-c3ea-44d8-bb0f-d6ccac6c7951 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:08:50.568402 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-38aa3633-c3ea-44d8-bb0f-d6ccac6c7951 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '36697cb548a14f47a84f7eebd452983f', 'request_id': 'req-38aa3633-c3ea-44d8-bb0f-d6ccac6c7951'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:50.574673 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-38aa3633-c3ea-44d8-bb0f-d6ccac6c7951 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '36697cb548a14f47a84f7eebd452983f', 'request_id': 'req-38aa3633-c3ea-44d8-bb0f-d6ccac6c7951'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:50.580767 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-38aa3633-c3ea-44d8-bb0f-d6ccac6c7951 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '36697cb548a14f47a84f7eebd452983f', 'request_id': 'req-38aa3633-c3ea-44d8-bb0f-d6ccac6c7951'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:50.593434 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-38aa3633-c3ea-44d8-bb0f-d6ccac6c7951 admin admin] The token cache is being invalidate because project 36697cb548a14f47a84f7eebd452983f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:08:50.593727 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-38aa3633-c3ea-44d8-bb0f-d6ccac6c7951 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:50.594341 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-38aa3633-c3ea-44d8-bb0f-d6ccac6c7951 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:08:50.595877 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1316/2603] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:08:50 2022] DELETE /identity/v3/projects/36697cb548a14f47a84f7eebd452983f => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:08:57.429177 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215ff8e2-581b-4bbd-91d2-6a42c8272c11 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:08:57.430583 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-215ff8e2-581b-4bbd-91d2-6a42c8272c11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:57.486976 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215ff8e2-581b-4bbd-91d2-6a42c8272c11 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:08:57.538305 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44704dd9-4bf2-431e-ae34-639d5168a30d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '993d553a46044c18a4eec6e408c0969a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:08:57.539087 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44704dd9-4bf2-431e-ae34-639d5168a30d service cinder] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:08:57.539364 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44704dd9-4bf2-431e-ae34-639d5168a30d service cinder] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:08:57.539661 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44704dd9-4bf2-431e-ae34-639d5168a30d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:08:57.541563 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-44704dd9-4bf2-431e-ae34-639d5168a30d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:57.601423 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44704dd9-4bf2-431e-ae34-639d5168a30d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:08:57.602969 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44704dd9-4bf2-431e-ae34-639d5168a30d service cinder] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:08:57.612222 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1288/2604] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:08:57 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:08:58.792259 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e40cb80-c1ce-4ecc-bfa3-c3d40b8df796 None None] REQUEST_METHOD: `POST` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:08:58.792773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e40cb80-c1ce-4ecc-bfa3-c3d40b8df796 None None] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:08:58.792773 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e40cb80-c1ce-4ecc-bfa3-c3d40b8df796 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:08:58.812097 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.auth.core [None req-1e40cb80-c1ce-4ecc-bfa3-c3d40b8df796 None None] MFA Rules not processed for user `f0b358738b5a46d8bc1db1da68040c8f`. Rule list: `[]` (Enabled: `True`). {{(pid=96565) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 31 15:08:58.813288 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1e40cb80-c1ce-4ecc-bfa3-c3d40b8df796 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:58.820497 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1317/2605] 10.222.0.36 () {58 vars in 958 bytes} [Mon Oct 31 15:08:58 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 31 15:08:59.573460 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44704dd9-4bf2-431e-ae34-639d5168a30d service cinder] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:08:59.574889 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-44704dd9-4bf2-431e-ae34-639d5168a30d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:59.649925 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44704dd9-4bf2-431e-ae34-639d5168a30d service cinder] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:08:59.654911 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9318d5a-86c2-4d6a-8270-01b38e9d7a82 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70ea2aaf529b46dabb70b71f4dcc3860', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:08:59.655510 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9318d5a-86c2-4d6a-8270-01b38e9d7a82 service placement] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:08:59.655821 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9318d5a-86c2-4d6a-8270-01b38e9d7a82 service placement] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:08:59.656097 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9318d5a-86c2-4d6a-8270-01b38e9d7a82 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:08:59.657832 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-b9318d5a-86c2-4d6a-8270-01b38e9d7a82 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:08:59.666923 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=100 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:08:59.669548 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9318d5a-86c2-4d6a-8270-01b38e9d7a82 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:08:59.671530 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9318d5a-86c2-4d6a-8270-01b38e9d7a82 service placement] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:08:59.681793 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1289/2606] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:08:59 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:09:08.807305 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=68,keystone:DELETE=12,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:09:09.677349 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:09:11.748749 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e40cb80-c1ce-4ecc-bfa3-c3d40b8df796 None None] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:09:11.751000 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-1e40cb80-c1ce-4ecc-bfa3-c3d40b8df796 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:09:11.829112 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e40cb80-c1ce-4ecc-bfa3-c3d40b8df796 None None] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:09:11.835018 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e49da3-3429-4c80-96fb-00fcd96d6c88 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8cace029a92406f850bb0a4ca07c461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'ResellerAdmin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:09:11.835652 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e49da3-3429-4c80-96fb-00fcd96d6c88 service nova] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:09:11.835931 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e49da3-3429-4c80-96fb-00fcd96d6c88 service nova] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:09:11.836204 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e49da3-3429-4c80-96fb-00fcd96d6c88 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:09:11.837960 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-f1e49da3-3429-4c80-96fb-00fcd96d6c88 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:09:11.850328 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e49da3-3429-4c80-96fb-00fcd96d6c88 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:09:11.851921 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e49da3-3429-4c80-96fb-00fcd96d6c88 service nova] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:09:11.862216 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1318/2607] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:09:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:09:21.857850 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=34 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:09:47.596407 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9318d5a-86c2-4d6a-8270-01b38e9d7a82 service placement] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:09:47.597799 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-b9318d5a-86c2-4d6a-8270-01b38e9d7a82 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:09:47.610879 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9318d5a-86c2-4d6a-8270-01b38e9d7a82 service placement] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:09:47.616011 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e82b23-bffa-404c-aea4-5a7b1932dde7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70ea2aaf529b46dabb70b71f4dcc3860', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:09:47.616595 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e82b23-bffa-404c-aea4-5a7b1932dde7 service placement] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:09:47.616872 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e82b23-bffa-404c-aea4-5a7b1932dde7 service placement] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:09:47.617141 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e82b23-bffa-404c-aea4-5a7b1932dde7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:09:47.618765 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-86e82b23-bffa-404c-aea4-5a7b1932dde7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:09:47.630419 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e82b23-bffa-404c-aea4-5a7b1932dde7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:09:47.632010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e82b23-bffa-404c-aea4-5a7b1932dde7 service placement] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:09:47.642046 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1290/2608] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:09:47 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:09:57.637500 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=6 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:11:00.075671 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e49da3-3429-4c80-96fb-00fcd96d6c88 service nova] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:11:00.077224 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-f1e49da3-3429-4c80-96fb-00fcd96d6c88 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:11:00.136247 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e49da3-3429-4c80-96fb-00fcd96d6c88 service nova] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:11:00.140070 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8a4342-be7c-47c2-ae89-0ccf0322d8e7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14946d97fd384e2a8bff68c4f53c2cfe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:11:00.140697 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8a4342-be7c-47c2-ae89-0ccf0322d8e7 service neutron] REQUEST_METHOD: `GET` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:11:00.140973 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8a4342-be7c-47c2-ae89-0ccf0322d8e7 service neutron] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:11:00.141268 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8a4342-be7c-47c2-ae89-0ccf0322d8e7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:11:00.143012 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-df8a4342-be7c-47c2-ae89-0ccf0322d8e7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:11:00.204973 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8a4342-be7c-47c2-ae89-0ccf0322d8e7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:11:00.206381 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8a4342-be7c-47c2-ae89-0ccf0322d8e7 service neutron] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:11:00.216610 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1319/2609] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:11:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:11:06.538287 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e82b23-bffa-404c-aea4-5a7b1932dde7 service placement] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:11:06.539850 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-86e82b23-bffa-404c-aea4-5a7b1932dde7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:11:06.555034 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e82b23-bffa-404c-aea4-5a7b1932dde7 service placement] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:11:06.559830 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b3d125-08fc-427c-a5b5-c8361a59f902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:11:06.560659 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b3d125-08fc-427c-a5b5-c8361a59f902 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:11:06.561120 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b3d125-08fc-427c-a5b5-c8361a59f902 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:11:06.561394 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b3d125-08fc-427c-a5b5-c8361a59f902 admin admin] PATH_INFO: `/v3/users/f0b358738b5a46d8bc1db1da68040c8f` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:11:06.562663 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b3d125-08fc-427c-a5b5-c8361a59f902 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f0b358738b5a46d8bc1db1da68040c8f)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:11:06.564623 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b3d125-08fc-427c-a5b5-c8361a59f902 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:11:06.613256 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-69b3d125-08fc-427c-a5b5-c8361a59f902 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f0b358738b5a46d8bc1db1da68040c8f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:06.620125 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-69b3d125-08fc-427c-a5b5-c8361a59f902 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f0b358738b5a46d8bc1db1da68040c8f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:06.620783 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-69b3d125-08fc-427c-a5b5-c8361a59f902 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f0b358738b5a46d8bc1db1da68040c8f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:06.621387 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-69b3d125-08fc-427c-a5b5-c8361a59f902 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f0b358738b5a46d8bc1db1da68040c8f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:06.633089 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-69b3d125-08fc-427c-a5b5-c8361a59f902 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f0b358738b5a46d8bc1db1da68040c8f'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:06.645281 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1291/2610] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:11:06 2022] DELETE /identity/v3/users/f0b358738b5a46d8bc1db1da68040c8f => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:11:06.812334 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8a4342-be7c-47c2-ae89-0ccf0322d8e7 service neutron] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:11:06.825656 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8a4342-be7c-47c2-ae89-0ccf0322d8e7 service neutron] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:11:06.882222 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65838c93-6f21-491e-8718-c9b1dc49c888 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:11:06.882855 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65838c93-6f21-491e-8718-c9b1dc49c888 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:11:06.883135 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65838c93-6f21-491e-8718-c9b1dc49c888 admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:11:06.883411 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65838c93-6f21-491e-8718-c9b1dc49c888 admin admin] PATH_INFO: `/v3/projects/d7f5a9331dd54d9e8b5033234cb1077a` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:11:06.884534 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65838c93-6f21-491e-8718-c9b1dc49c888 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d7f5a9331dd54d9e8b5033234cb1077a)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:11:06.886410 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65838c93-6f21-491e-8718-c9b1dc49c888 admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:11:06.932472 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-65838c93-6f21-491e-8718-c9b1dc49c888 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd7f5a9331dd54d9e8b5033234cb1077a', 'request_id': 'req-65838c93-6f21-491e-8718-c9b1dc49c888'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:06.934407 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:11:06.943210 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-65838c93-6f21-491e-8718-c9b1dc49c888 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd7f5a9331dd54d9e8b5033234cb1077a', 'request_id': 'req-65838c93-6f21-491e-8718-c9b1dc49c888'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:06.949441 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-65838c93-6f21-491e-8718-c9b1dc49c888 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd7f5a9331dd54d9e8b5033234cb1077a', 'request_id': 'req-65838c93-6f21-491e-8718-c9b1dc49c888'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:06.961650 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-65838c93-6f21-491e-8718-c9b1dc49c888 admin admin] The token cache is being invalidate because project d7f5a9331dd54d9e8b5033234cb1077a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:11:06.961940 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-65838c93-6f21-491e-8718-c9b1dc49c888 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:06.962560 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-65838c93-6f21-491e-8718-c9b1dc49c888 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:06.964141 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1320/2611] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:11:06 2022] DELETE /identity/v3/projects/d7f5a9331dd54d9e8b5033234cb1077a => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:11:06.968111 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b3d125-08fc-427c-a5b5-c8361a59f902 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:11:06.969503 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-69b3d125-08fc-427c-a5b5-c8361a59f902 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:11:06.983356 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b3d125-08fc-427c-a5b5-c8361a59f902 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:11:07.042010 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab3e0d6-c5df-43c8-98c0-bdcb3f92f292 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:11:07.042563 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab3e0d6-c5df-43c8-98c0-bdcb3f92f292 admin admin] REQUEST_METHOD: `DELETE` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:11:07.042847 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab3e0d6-c5df-43c8-98c0-bdcb3f92f292 admin admin] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:11:07.043129 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab3e0d6-c5df-43c8-98c0-bdcb3f92f292 admin admin] PATH_INFO: `/v3/users/5324a9ce10ae4460ae36bac43f35c1c6` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:11:07.044390 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab3e0d6-c5df-43c8-98c0-bdcb3f92f292 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5324a9ce10ae4460ae36bac43f35c1c6)` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:11:07.046315 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab3e0d6-c5df-43c8-98c0-bdcb3f92f292 admin admin] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:11:07.092637 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0ab3e0d6-c5df-43c8-98c0-bdcb3f92f292 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5324a9ce10ae4460ae36bac43f35c1c6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:07.100654 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0ab3e0d6-c5df-43c8-98c0-bdcb3f92f292 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5324a9ce10ae4460ae36bac43f35c1c6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:07.101314 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0ab3e0d6-c5df-43c8-98c0-bdcb3f92f292 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5324a9ce10ae4460ae36bac43f35c1c6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:07.101923 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0ab3e0d6-c5df-43c8-98c0-bdcb3f92f292 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5324a9ce10ae4460ae36bac43f35c1c6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:07.115068 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.notifications [None req-0ab3e0d6-c5df-43c8-98c0-bdcb3f92f292 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5324a9ce10ae4460ae36bac43f35c1c6'} {{(pid=96564) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:07.116941 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=80,keystone:DELETE=19,keystone:INSERT=1 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:11:07.129786 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1292/2612] 10.222.0.36 () {58 vars in 1224 bytes} [Mon Oct 31 15:11:06 2022] DELETE /identity/v3/users/5324a9ce10ae4460ae36bac43f35c1c6 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:11:07.304131 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65838c93-6f21-491e-8718-c9b1dc49c888 admin admin] Authenticating user token {{(pid=96565) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:11:07.305573 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.fernet_utils [None req-65838c93-6f21-491e-8718-c9b1dc49c888 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96565) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:11:07.321461 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65838c93-6f21-491e-8718-c9b1dc49c888 admin admin] Validating token access rules against request {{(pid=96565) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:11:07.380011 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd181629-14ca-4296-8186-c8b2670fc3eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c12195059ec40bfafbb7be8e9f46734', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e18b2b57837347e48db3adc0214764b8', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96565) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:11:07.380648 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd181629-14ca-4296-8186-c8b2670fc3eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:11:07.380938 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd181629-14ca-4296-8186-c8b2670fc3eb admin admin] SCRIPT_NAME: `/identity` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:11:07.381215 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd181629-14ca-4296-8186-c8b2670fc3eb admin admin] PATH_INFO: `/v3/projects/f1c52c7d41be4e1aa934c498760d7403` {{(pid=96565) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:11:07.382335 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd181629-14ca-4296-8186-c8b2670fc3eb admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1c52c7d41be4e1aa934c498760d7403)` {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:11:07.384272 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd181629-14ca-4296-8186-c8b2670fc3eb admin admin] RBAC: Authorization granted {{(pid=96565) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:11:07.429948 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-cd181629-14ca-4296-8186-c8b2670fc3eb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1c52c7d41be4e1aa934c498760d7403', 'request_id': 'req-cd181629-14ca-4296-8186-c8b2670fc3eb'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:07.435397 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-cd181629-14ca-4296-8186-c8b2670fc3eb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1c52c7d41be4e1aa934c498760d7403', 'request_id': 'req-cd181629-14ca-4296-8186-c8b2670fc3eb'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:07.441643 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-cd181629-14ca-4296-8186-c8b2670fc3eb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1c52c7d41be4e1aa934c498760d7403', 'request_id': 'req-cd181629-14ca-4296-8186-c8b2670fc3eb'} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:07.454414 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-cd181629-14ca-4296-8186-c8b2670fc3eb admin admin] The token cache is being invalidate because project f1c52c7d41be4e1aa934c498760d7403 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=96565) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 31 15:11:07.454707 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-cd181629-14ca-4296-8186-c8b2670fc3eb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:07.455358 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG keystone.notifications [None req-cd181629-14ca-4296-8186-c8b2670fc3eb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=96565) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 31 15:11:07.457714 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: [pid: 96565|app: 0|req: 1321/2613] 10.222.0.36 () {58 vars in 1233 bytes} [Mon Oct 31 15:11:07 2022] DELETE /identity/v3/projects/f1c52c7d41be4e1aa934c498760d7403 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 31 15:11:17.128139 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:INSERT=1,keystone:DELETE=1,keystone:SELECT=2 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:11:17.450983 dv-cinder-860227-3-iscsi devstack@keystone.service[96565]: DEBUG dbcounter [-] [96565] Writing DB stats keystone:SELECT=57,keystone:DELETE=9,keystone:INSERT=2 {{(pid=96565) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 31 15:14:52.549195 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab3e0d6-c5df-43c8-98c0-bdcb3f92f292 admin admin] Authenticating user token {{(pid=96564) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 31 15:14:52.550640 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-0ab3e0d6-c5df-43c8-98c0-bdcb3f92f292 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:14:52.686534 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab3e0d6-c5df-43c8-98c0-bdcb3f92f292 admin admin] Validating token access rules against request {{(pid=96564) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 31 15:14:52.746409 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df8bd30-0254-4fac-9dd1-f76abaa8f564 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70ea2aaf529b46dabb70b71f4dcc3860', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4118e33b099f4cf0a3e3a46b211ba1d8', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=96564) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 31 15:14:52.747012 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df8bd30-0254-4fac-9dd1-f76abaa8f564 service placement] REQUEST_METHOD: `GET` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 31 15:14:52.747291 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df8bd30-0254-4fac-9dd1-f76abaa8f564 service placement] SCRIPT_NAME: `/identity` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 31 15:14:52.747584 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df8bd30-0254-4fac-9dd1-f76abaa8f564 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=96564) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 31 15:14:52.749328 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.fernet_utils [None req-9df8bd30-0254-4fac-9dd1-f76abaa8f564 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=96564) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 31 15:14:52.760947 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df8bd30-0254-4fac-9dd1-f76abaa8f564 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 31 15:14:52.762515 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df8bd30-0254-4fac-9dd1-f76abaa8f564 service placement] RBAC: Authorization granted {{(pid=96564) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 31 15:14:52.772402 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: [pid: 96564|app: 0|req: 1293/2614] 10.222.0.36 () {60 vars in 1365 bytes} [Mon Oct 31 15:14:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 31 15:15:02.768050 dv-cinder-860227-3-iscsi devstack@keystone.service[96564]: DEBUG dbcounter [-] [96564] Writing DB stats keystone:SELECT=89 {{(pid=96564) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}